Custom URL matching on wrong URLs

[mprintz]

Hi,

I have a security rule that's supposed to be only allowing traffic for URLs in a custom URL category.  However, it appears that it's matching lots of other URLs that aren't in the category.  Below are some screenshots.  I'm running v8.0.6.  Let me know what other info you might need and what I'm doing wrong.  Thanks.

Matt

Security rule showing URL category specified:

Custom URL category:

Unified log showing other URLs not listed above matching security rule (e.g. first and last lines):

[kiwi]

Hi @mprintz,

I believe something went wrong when uploading the screenshots.

Cheers !

-Kiwi.

[mprintz]

Sorry about that, not sure what happened.  I could even see them when I did the preview.  Let's try again.

Matt

Security rule:

Custom URL category:

Log - in this case, lines 3, 6, & 11 showing the mismatched URLs:

[OtakarKlier]

Hello,

I'm not able to see the images.

Regards,

[mprintz]

Wow, I don't know what's going on here with the pictures.  I see them after posting, but then a few minutes later they don't show up.  Let's try it via links instead of embedded photos.

Matt

Security Rule: http://www.harmelin.com/Palo/secrule.png

URL Category: http://www.harmelin.com/Palo/urlcat.png

Log: http://www.harmelin.com/Palo/log.png

[BPry · Accepted solution]

@mprintz,

This is usually caused because you've allowed akamai.net as a URL. You only actually need to allow the following. 

• http://windowsupdate.microsoft.com
  
  
• http://*.windowsupdate.microsoft.com
  
  
• https://*.windowsupdate.microsoft.com
  
  
• http://*.update.microsoft.com
  
  
• https://*.update.microsoft.com
  
  
• http://*.windowsupdate.com
  
  
• http://download.windowsupdate.com
  
  
• http://download.microsoft.com
  
  
• http://*.download.windowsupdate.com
  
  
• http://test.stats.update.microsoft.com
  
  
• http://ntservicepack.microsoft.com

[Rags]

As BPry said above, the 2 bottom IPs are probably going through something hosted by akamai. I see them listed as zscaler, they might be a cloud platform hosted on akamai. 

https://technet.microsoft.com/en-us/library/bb693717.aspx

Same list as above

[BPry]

@Rags,

Funny enough while ZScaler is a competitor of some of the features that Akamai provides; they actually utilize akamai for ZEN lookup. 

[Remo]

@mprintz

As proposed by @BPry and @Rags you should change the url category.

But did you also filter for the destination IPs of these 2 log entries. As this log shows a threat log entry with the subtype spyware these entries don't necessarily mean that this traffic was allowed - 3 log entries aren't enough to say that for sure. It could be that the antispyware feature logged this but after that the traffic was blocked because of the url not matching your custom url category.

(Do you log the theat for TLS evasions? Are these 2 entries such threats?)

[mprintz]

I changed the Custom URL Category so that it only contains the URLs below, but I'm still seeing all sorts of other URLs in the logs (http://www.harmelin.com/Palo/log2.png).  Any other ideas?  Thanks.

@BPry We are using Zscaler, so it would make sense that some of the traffic would have been hitting that rule before, if it was destined for akamai and Zscaler uses their service.

Matt

*.update.microsoft.com

*.windowsupdate.com

*.windowsupdate.microsoft.com

download.microsoft.com

ntservicepack.microsoft.com

windowsupdate.microsoft.com

[BPry]

@mprintz,

Are you decrypting traffic at all or not? 

[mprintz]

@BPry Yes, I am for that URL category and a handful of other URLs (none of the ones in the logs I've posted), as well as some non-HTTP services.  However, that's a recent change I made to see if it would fix the problem, but it didn't.

Matt

[Remo · Accepted solution]

Hi @mprintz

What you actually have in this screenshot are connection attempts. In the screenshot there are no urls in the URL column in the monitor tab, so the firewall was not able to apply the url category. But this does not mean that these connections were successfullly established (also because of the app incomplete - I assume the bytes (received/sent) are only a few, not much more that a tcp handshake and a tls handshake).

The firewall has to allow some packets in order to get to the packet where it could allow/deny the traffic based on the actual url.

[mprintz · Accepted solution]

@Remo Thanks, that makes sense.  I didn't realize the handshake is considered a different session than the data that follows it.  I also moved the rule down in the list (as it's not as frequently used as others) so other rules are hit first.