Creating a global, URL based whitelist rule

[jessica-davis]

I'm trying to build a global rule for Sophos cloud based services. I've built a list of all the URLs they use, added the URL list to the URL category part of the rule with the applications web-browsing, ssl, sophos-update and sophos-live-protection, and generally it seemed to work with a small snag. I noticed a LOT of traffic was hitting this rule for the first few packets, until the PAN could determine it wasn't going to one of the listed URLs, then either passing to another rule, or ending the session. 

The problem is, we have systems that have highly limited access to the internet, so I want to restrict this down somehow. Unfortunately, Sophos uses multiple CDNs for distribution with short TTLs on the DNS, so the IPs can change minute to minute, and generating a complete list of IPs isn't really feasible.

Is there a good way to make a single rule for 'allow everyone inside going to this URL/URL category' without catching basically all internet bound traffic?

[JoeAndreini]

what about a custom application?

Use the hostnames from your URLs to match in the http-req-host-header and the paths (if needed) to match http-req-uri-path, that way this traffic will be able to pass to another rule if it does not match the application.

URL filtering is not a match criteria to determine what rule applies to a session.

[BPry]

@jessica-davis,

I think @JoeAndreini's answer is about as good as you could hope for with what you are trying to accomplish. However this would only really work if you have already implemented SSL Decryption. CDNs get near impossible to work with if you can't see the entire request. 

[Remo · Accepted solution]

@JoeAndreini this actually is another way to do what @jessica-davis already did. Here it depends on what you prefer. If you use the URL category directly in your security rule and not with an URL profile it is a filtering criteria, but because the firewall needs some packets to get to the http request/tls client hello this rule will also allow the first packets of other connections.

@jessica-davisthis rule does not allow any connection towards the internet, but as I wrote that the firewall need to allow some packets to bo able to filter on URLs, some packets need to be allowed. So filtering based on URLs (no matter if you create an app or do it the way you already did) will always have this "sideeffect". And as soon the firewall sees that the connection does not match one of the URLs in your custom category, it evaluates the ruleset again to check if there is another match for this connection and probably dropps the connection as I assume there is no other rule because you want to restrict the internetacces as much as possible.

[Remo]

@BPry

You are absolutely right when you need to filter on parts of the URL after the FQDN, but if not this works in almost all cases without having decryptiom enabled. This is at least my experience with applying URL filtering without TLS decryption (--> SNI/hostname extension in the TLS client hello packet)

[BPry]

@Remo,

If I recall correctly with Sophos CDN you actually do need to filter on parts of the URL and not strictly what the firewall can see in the SNI. 

[Remo]

Ok, then it is clear 😉

[jessica-davis]

Thank you for that. That's what I was thinking as I saw the behavior, was just hoping to avoid the rule logging so much just...random noise. I'm trying a few other things, namely pulling a list of the various destination host names being hit and hoping I can build a list of FQDN address objects to mitigate it slightly. I have to straighten out some DNS issues before that's a viable option, though. Thank you for the suggestions!