topic Re: Custom URL matching on wrong URLs in General Topics

Custom URL matching on wrong URLs

mprintz — Tue, 27 Feb 2018 21:13:48 GMT

Hi,

I have a security rule that's supposed to be only allowing traffic for URLs in a custom URL category. However, it appears that it's matching lots of other URLs that aren't in the category. Below are some screenshots. I'm running v8.0.6. Let me know what other info you might need and what I'm doing wrong. Thanks.

Matt

Security rule showing URL category specified:

Custom URL category:

Unified log showing other URLs not listed above matching security rule (e.g. first and last lines):

Re: Custom URL matching on wrong URLs

kiwi — Wed, 28 Feb 2018 08:14:58 GMT

Hi @mprintz,

I believe something went wrong when uploading the screenshots.

Cheers !

-Kiwi.

Re: Custom URL matching on wrong URLs

mprintz — Wed, 28 Feb 2018 13:36:51 GMT

Sorry about that, not sure what happened. I could even see them when I did the preview. Let's try again.

Matt

Security rule:

Custom URL category:

Log - in this case, lines 3, 6, & 11 showing the mismatched URLs:

Re: Custom URL matching on wrong URLs

OtakarKlier — Wed, 28 Feb 2018 15:00:42 GMT

Hello,

I'm not able to see the images.

Regards,

Re: Custom URL matching on wrong URLs

mprintz — Wed, 28 Feb 2018 15:18:58 GMT

Wow, I don't know what's going on here with the pictures. I see them after posting, but then a few minutes later they don't show up. Let's try it via links instead of embedded photos.

Matt

Security Rule: http://www.harmelin.com/Palo/secrule.png

URL Category: http://www.harmelin.com/Palo/urlcat.png

Log: http://www.harmelin.com/Palo/log.png

Re: Custom URL matching on wrong URLs

BPry — Wed, 28 Feb 2018 17:52:05 GMT

@mprintz,

This is usually caused because you've allowed akamai.net as a URL. You only actually need to allow the following.

http://windowsupdate.microsoft.com
http://*.windowsupdate.microsoft.com
https://*.windowsupdate.microsoft.com
http://*.update.microsoft.com
https://*.update.microsoft.com
http://*.windowsupdate.com
http://download.windowsupdate.com
http://download.microsoft.com
http://*.download.windowsupdate.com
http://test.stats.update.microsoft.com
http://ntservicepack.microsoft.com

Re: Custom URL matching on wrong URLs

Rags — Wed, 28 Feb 2018 19:15:23 GMT

As BPry said above, the 2 bottom IPs are probably going through something hosted by akamai. I see them listed as zscaler, they might be a cloud platform hosted on akamai.

https://technet.microsoft.com/en-us/library/bb693717.aspx

Same list as above

Re: Custom URL matching on wrong URLs

BPry — Wed, 28 Feb 2018 19:25:34 GMT

@Rags,

Funny enough while ZScaler is a competitor of some of the features that Akamai provides; they actually utilize akamai for ZEN lookup.

Re: Custom URL matching on wrong URLs

Remo — Wed, 28 Feb 2018 21:00:27 GMT

@mprintz

As proposed by @BPry and @Rags you should change the url category.

But did you also filter for the destination IPs of these 2 log entries. As this log shows a threat log entry with the subtype spyware these entries don't necessarily mean that this traffic was allowed - 3 log entries aren't enough to say that for sure. It could be that the antispyware feature logged this but after that the traffic was blocked because of the url not matching your custom url category.

(Do you log the theat for TLS evasions? Are these 2 entries such threats?)

Re: Custom URL matching on wrong URLs

mprintz — Wed, 28 Feb 2018 21:19:34 GMT

I changed the Custom URL Category so that it only contains the URLs below, but I'm still seeing all sorts of other URLs in the logs (http://www.harmelin.com/Palo/log2.png). Any other ideas? Thanks.

@BPry We are using Zscaler, so it would make sense that some of the traffic would have been hitting that rule before, if it was destined for akamai and Zscaler uses their service.

Matt

*.update.microsoft.com

*.windowsupdate.com

*.windowsupdate.microsoft.com

download.microsoft.com

ntservicepack.microsoft.com

windowsupdate.microsoft.com

Re: Custom URL matching on wrong URLs

BPry — Wed, 28 Feb 2018 21:21:58 GMT

@mprintz,

Are you decrypting traffic at all or not?

Re: Custom URL matching on wrong URLs

mprintz — Wed, 28 Feb 2018 21:29:11 GMT

@BPry Yes, I am for that URL category and a handful of other URLs (none of the ones in the logs I've posted), as well as some non-HTTP services. However, that's a recent change I made to see if it would fix the problem, but it didn't.

Matt

Re: Custom URL matching on wrong URLs

Remo — Wed, 28 Feb 2018 21:30:26 GMT

Hi @mprintz

What you actually have in this screenshot are connection attempts. In the screenshot there are no urls in the URL column in the monitor tab, so the firewall was not able to apply the url category. But this does not mean that these connections were successfullly established (also because of the app incomplete - I assume the bytes (received/sent) are only a few, not much more that a tcp handshake and a tls handshake).

The firewall has to allow some packets in order to get to the packet where it could allow/deny the traffic based on the actual url.

Re: Custom URL matching on wrong URLs

mprintz — Thu, 01 Mar 2018 13:59:49 GMT

@Remo Thanks, that makes sense. I didn't realize the handshake is considered a different session than the data that follows it. I also moved the rule down in the list (as it's not as frequently used as others) so other rules are hit first.