URL filtering not working

[nwnetadmin]

My issue is that the url filtering isn't working. I for example, I can browse to

urlfiltering.paloaltonetworks.com/test-adult and it isn't blocked.

This is on a PA-220. It is currently running 10.1.3-h3. Earlier today, I noticed that the URL filtering license was expired, but I just did "retrieve license keys from license server" and now it shows that it si good. Both the "PAN-DB URL Filtering" and "Advanced URL Filtering" has valid licenses. However, I haven't rebooted since then.

One thing that I noticed is that, the url filter logs, it shows:

I tracked the allow-web-browsing rule and that points to the outbound-url security profile which clearly blocks the adult category.

So, it is categorizing it wrong. I wonder if that is related to the fact that the URL is only showing the host name portion. Like, it isn't considering the /test-adult/ and that is why it can't put it in the right category? If that is by design, then it was defeat the point of having the test sites.

I did the below commands. 

admin@PA-220> show url-cloud status

PAN-DB URL Filtering
License : valid
Current cloud server : serverlist3.urlcloud.paloaltonetworks.com
Cloud connection : connected
Cloud mode : public
URL database version - device : 20240222.20354
URL database version - cloud : 20240222.20354 ( last update time 2024/02 15:32:13 )
URL database status : good
URL protocol version - device : pan/2.0.0
URL protocol version - cloud : pan/2.0.0
Protocol compatibility status : compatible

admin@PA-220> test url http://urlfiltering.paloaltonetworks.com/test-adult

urlfiltering.paloaltonetworks.com/test-adult not-resolved (Base db) mlav_flagexpires in 5 seconds
urlfiltering.paloaltonetworks.com/test-adult adult low-risk (Cloud db)

Is it possible that it isn't "digesting" the Cloud db properly and not applying it to the Base db?

I am going to update to 10.1.11-h5 tonight. Maybe that will help. Maybe a reboot since the license issue is resolved will help. In the meantime, I thought I'd put it out there and see if anyone else had any better ideas. I might open a case with support.

[iarobertson]

Hello @nwnetadmin - I would suggest a TAC case is appropriate here. 

[nwnetadmin]

@iarobertson thanks. I did open a case last night. We wound up creating a decryption profile, which created other issues.

Incidentally, they said that the decryption was necessary to read sni values.

My question is.... is it necessary to use decryption for URL filtering? ie for correct categorization. I was using the pan test sites to verify url filtering. But, I wonder if that presented the problem a problem that isn't necessarily a problem with the actual sites to be blocked.

The palo test sites all use the same host portion of the URL, it's just the /test-hacking/ portion that designates the category. But, in order for the firewall to know the difference between the two, it needs to look at the whole path, and maybe it can't do that without actual decryption?

But actual hacking sites that we are actually trying to block would use different hostnames. just like yahoo.com is different than google.com. So maybe the url filtering would work better, without decryption, in real world scenarios?

Now, I can appreciate that a more comprehensive security posture analyze the whole path. Within this conversation, I am just trying to better understand.

[Claw4609 · Accepted solution]

"My question is.... is it necessary to use decryption for URL filtering? ie for correct categorization."

For fully correct categorization, yes. Without decryption you'll just see the main domain, i.e. google.com/ vs google.com/something

Generally ssl decryption for url filtering is not necessarily the biggest deal, except in these test cases. Dont get me wrong, it could be, its just a really small sample. With the example of the palos site, basically everything is going to be computer-and-internet-info with the exception being those test pages

Having ssl inspection enabled is also Palo Alto best practices due to inspecting traffic, application identifications (and by extenstion more granular controls), and things like full url identification and credential detections.  

Test URL Filtering Configuration (paloaltonetworks.com)

[rdumoulin · Accepted solution]

---

@nwnetadmin wrote:

@iarobertson thanks. I did open a case last night. We wound up creating a decryption profile, which created other issues.

Incidentally, they said that the decryption was necessary to read sni values.

My question is.... is it necessary to use decryption for URL filtering? ie for correct categorization. I was using the pan test sites to verify url filtering. But, I wonder if that presented the problem a problem that isn't necessarily a problem with the actual sites to be blocked.

The palo test sites all use the same host portion of the URL, it's just the /test-hacking/ portion that designates the category. But, in order for the firewall to know the difference between the two, it needs to look at the whole path, and maybe it can't do that without actual decryption?

But actual hacking sites that we are actually trying to block would use different hostnames. just like yahoo.com is different than google.com. So maybe the url filtering would work better, without decryption, in real world scenarios?

Now, I can appreciate that a more comprehensive security posture analyze the whole path. Within this conversation, I am just trying to better understand.

---

Hello @nwnetadmin 

 You are right about the host portion of urlfiltering.paloaltonetworks.com/test-adult. 

If you capture the ssl handshake with wireshark, you will see in the ssl client hello that the SNI is urlfiltering.paloaltonetworks.com. 

It is the same for the CN in the certificate; the CN is urlfiltering.paloaltonetworks.com. 

So there is no way for the URL filtering profile to categorize the URL unless the traffic is decrypted. 

On the other hand, I tested with another adult site where the SNI clearly indicated adult content, and the URL filter correctly categorized the site as adult, and therefore the access was "block-url". 

regards

--Richard

[nwnetadmin]

thanks to both of you.