Custom URL category with directories

[chens]

Hi guys,

I am trying to create a custom url category to allow only these (s3.amazonaws.com/icount-pdfs😞

example:

https://s3.amazonaws.com/icount-pdfs/57764_25566fbb6fd6bbab6b0f35eba91bb55e.pdf?17016197031

i have tried:  

s3.amazonaws.com/icount-pdfs/*

s3.amazonaws.com/icount-pdfs

s3.amazonaws.com/icount-pdfs/

None of these works.

ideas?

[Adrian_Jensen · Accepted solution]

You should be able to create a single custom URL category to match that:

s3.amazonaws.com/icount-pdfs/

Though note you will have to be doing SSL decryption to identify that path as the PA can't see the actual path unless the SSL/TLS is decrypted.

[LBSalvat]

I have a similar issue where I have a rule that I'm trying to limit allowed traffic with custom URL category that includes 3 hostnames with wildcards. The rule is hit when the traffic hits 2 of the hostnames but my third entry (*.amazonaws.com) keeps hitting the default rules. What's interesting is that in the monitor tab, if I click the "resolve hostnames" checkbox, the traffic for 2 hostnames that do work wont resolve while the traffic that hits the default resolves to the amazonaws host.

Thoughts?

[Adrian_Jensen]

There can be several issues and URL categories can be tricky to implement with how many different ways traffic can be passed. It is important to remember that URL categories filter based on the FQDN/URL contained within a HTTP/HTTPS (or similar) connection (IP layer 5-7), not based on the DNS name of the destination IP (IP layer 3). From your description I can think of several different possibilities:

1) The "Resolve hostname" checkbox in  Monitor->Logs->Traffic does not show the hostname connected to in the HTTP/HTTPS request. It returns the current reverse-DNS response for the IPs shown in the logs. When the client originally looked up a DNS name it resolved to an IP which is hosting that site, as well as possibly hundreds/thousands of other sites at the same time. The rDNS name may have no relation to the originally called name (i.e. "www.example.com" is hosted on AmazonAWS and resolves to 203.0.113.58, but 203.0.113.58 reverse resolves to "s3-a589.r5-west.srv.amazonaws.com").

2) You can also use URL Categories for filtering just based on the host name (i.e. "example.com/") without SSL/TLS decryption up to TLS1.2, and to a lesser extent TLS1.3 (depending on the PA version and how TLS1.3 connected). This uses hints in the SSL/TLS certificate exchange (SNI - Server Name Indication) before the session is encrypted. However, in order to use URL Categories with a host directory path (i.e "example.com/naughty/files/"), you must be running SSL/TLS decryption as the path name is within the encrypted session (or only looking at non-encrypted HTTP traffic). 

3) Under certain circumstances, the client may connect to a host that responds with a SSL certificate that covers multiple hostnames. The client might reuse that existing SSL session to connect to the alternate host name that would otherwise be blocked, but if you are not running SSL/TLS decryption you can't see the target hostname change. (i.e. you block "forum.example.com/" but allow other "*.example.com/" traffic, the client connects with a request for "www.example.com" [which the PA sees and allowed], the server responds with a certificate for "www.example.com" and additional server names "images.example.com" and "forum.example.com" [which is in the encrypted certificate exchange], if the client now reuses the existing SSL session to now request "forum.example.com" files from the same host the PA doesn't know the hostname has changed.)