URL Blank in Traffic Logs

[ALSdburns]

The traffic logs for our PAs almost never actually show a URL, despite the URL category getting properly assigned. The only time I ever see a URL show up in the logs is if it is specifically denied because of the URL category, which is fairly rare. If they are allowed, or blocked based on something else like application, no URL shows. Is this expected behavior or could it be a configuration issue on our end? I am a bit confused by this as adding a URL to a custom URL category is successful in allowing specific sites, so the PA is obviously aware of the URL, but it doesn't show in the logs. A common scenario is a site being blocked based on application but we need to make an exception for a specific URL. I can't grab the URL from the logs and will instead need to contact the user to gather that information. I will then add that URL to a custom URL category, put that in a rule with the relevant application and the URL will be allowed. Throughout that whole process, the logs will not show the URL in the traffic logs despite the PA allowing based on the URL category. All of the relevant policies have a URL Filtering profile applied and log at the end of the session. 

[TomYoung · Accepted solution]

Hi @ALSdburns ,

The default action of allow for URLs does NOT log the URL.  If you want to log the URL, set the action to alert.  https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/url-filtering/url-categories/url-filtering...

Thanks,

Tom

[ALSdburns]

Thank you for the info! Exactly what I was looking for.

[ALSdburns]

Sorry to bring this up again, but after testing the alert setting I had one follow up question. How would we get the URL to show for traffic that was blocked based on other factors, such as application? 

I imagine I should just apply our default URL filtering to our Block All rule, but want to doublecheck. I assume even if you "allow" certain categories in the applied URL Filter, it will still get blocked if the actual rule is a Deny rule?

[TomYoung]

Hi @ALSdburns ,

That is a great question.  I cannot find any documentation for that scenario.  I can understand if traffic is being blocked that you may want to see the URL in order to determine if it should be allowed.

I can only assume that once the NGFW determines that the traffic will be dropped, it stops the session setup and never gets to the URL lookup.

One idea is to resolve the destination IP (manually or by Resolve hostname on the bottom of the Monitor tab) in order to find the parent domain.

Thanks,

Tom

[Adrian_Jensen]

In addition to what @TomYoung says, a single session may also contain many, many URLs that were allowed prior to the session being blocked for an unrelated reason (App-ID, etc.). When a device opens a connection it can request a URL and receive a response. Before or after that response is received, the client can make another request over the same session. This is called HTTP pipelining, where multiple resources can be inflight at the same time. If there are multiple URLs which are allowed, which URL would you log? In other cases it may not even be HTTP communication, so there is no URL associated with the Application ID.

A second way to possibly resolve a hostname for the blocked site is to look at your PA DNS cache (if you are resolving/filtering internal DNS off your PA). From the CLI, "show dns-proxy cache all | match <ip address>" will show you any matching FQDNs or the IP.