I do daily scripted syslog reports for traffic through the firewall. PA syslog messages are pretty good actually. However, the only messages that have something that resembles URL is the messages of the "THREAT url" pattern. Now, I accasionally need to do a web usage report for some managers on what their employees are doing. For that I need full URLs for the session along with more information, eg user agent, type of HTML request (get/post) etc. Not just for THREAT pattern, but for the normal traffic pattern (TRAFFIC start/end) as well. PA capable of that at all?
I would recommend installing a SIEM anyways if you don't already have one if you are interacting with Syslog at all already. Graylog (Free/Paid) or Splunk (Paid) would be my go-to for something like this.
Your request isn't entirely clear but does your firewall actually log the URL information for all of your traffic to begin with, or not? In most environments you wouldn't actually be setup to log every visited URL from your firewall, which would diminish the effectfullness of the requested report. You'd want to look at your URL logs on the firewall and ensure you are actually logging the accessed URL for the traffic you wish to log, and possibly make some configuration changes to your URL FIltering profile if you are not seeing the logs you require recorded on the firewall.
Hi, thank you for your reply. I really appriciate the help.
I already have the syslog server and the siem configured and all is well in that regard. Syslog is recieveing fine and logs are really easy to read and very well structured.
The problem I'm having is the web usage reporting. Because now we no longer have the internet proxy (which used to log users' activity) I need to get that information from the syslog messages coming from the PA.
If I search for the "TRAFFIC,start" (or TRAFFIC,end) pattern, there is no destination URL in the syslog message line. Only destination IP.
If I search for the "THREAT,url" patterrn, I get the destination URL and can do some sort of web usage reporting.
As I understand all webfiltering activity logs as "THREAT,url". However, it only llogs if the action is set to anything but "allow". When action is set to "allow", PA logs nothing at at all. Therefore we had to set all to "alert"
The TRAFFIC,start messages contain plenty of https (443) traffic, but do not log URLs. I was wandering if the TREAT message are included in the TRAFFIC messages (do they double up?) or the TRAFFIC messages include tflows that are not filtered for some reason. Below are the stats for a day. And the numbers just do not stack up.
4076679 TRAFFIC end
3468509 TRAFFIC start
1235392 THREAT url
483048 TRAFFIC drop
2103 TRAFFIC deny
517 THREAT vulnerability
5 THREAT spyware
So the URL logs are a completely seperate thing; they aren't located in the Traffic or Threat logs unless it's being recorded because a threat or something of the like was identified.
You actually want to ensure that the firewall is configured to send the URL logs to your SIEM and that the URL Filtering profile assigned to outbound traffic is set to at least 'alert' on every single category so the URL actually gets recorded. You then would have to utilize an extractor to record the actual fields within the message to the proper tag and then build your report in correlation with the Traffic logs and the URL logs if you want a report similar to what your proxy was likely building.
yes we did that. We set all categories to "alert" so we are recording URLs for all categories. When searching the syslog file for the "inside,outside" and "THREAT,url" I get the URLs in field 32 fine. Extracting it and matching to source IP is really easy.
however, there is a lot of traffic going through the firewall with the "inside,outside" and "TRAFFIC,start" pattern. And lots of it on 80 and 443. Logically all that traffic should match the URL Filtering profile, right? Does that mean we have overlapping rules that do not have URL Filtering profile applied to them?
From your comment I reason that all "inside,outside" should match "THREAT,url" and there should not be "TRAFFIC,start" (or end) Am I missing correct? logically. Or am I missing somthing.
You appear to be looking at just two of the log databases when you filter your syslog messages, the Threat and Traffic databases.
There is a completely separate set of logs on the firewall in the URL database. This is where all of the URLs visited are actually located.
finally after a lot of searching I found it. URL logs are, in fact, exported as part of the THREAT string. Only they are reported as THREAT,url
it's really strange. I don't know why PA do it that way, but it is what it is
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!