I do daily scripted syslog reports for traffic through the firewall. PA syslog messages are pretty good actually. However, the only messages that have something that resembles URL is the messages of the "THREAT url" pattern. Now, I accasionally need to do a web usage report for some managers on what their employees are doing. For that I need full URLs for the session along with more information, eg user agent, type of HTML request (get/post) etc. Not just for THREAT pattern, but for the normal traffic pattern (TRAFFIC start/end) as well. PA capable of that at all?
I would recommend installing a SIEM anyways if you don't already have one if you are interacting with Syslog at all already. Graylog (Free/Paid) or Splunk (Paid) would be my go-to for something like this.
Your request isn't entirely clear but does your firewall actually log the URL information for all of your traffic to begin with, or not? In most environments you wouldn't actually be setup to log every visited URL from your firewall, which would diminish the effectfullness of the requested report. You'd want to look at your URL logs on the firewall and ensure you are actually logging the accessed URL for the traffic you wish to log, and possibly make some configuration changes to your URL FIltering profile if you are not seeing the logs you require recorded on the firewall.
Hi, thank you for your reply. I really appriciate the help.
I already have the syslog server and the siem configured and all is well in that regard. Syslog is recieveing fine and logs are really easy to read and very well structured.
The problem I'm having is the web usage reporting. Because now we no longer have the internet proxy (which used to log users' activity) I need to get that information from the syslog messages coming from the PA.
If I search for the "TRAFFIC,start" (or TRAFFIC,end) pattern, there is no destination URL in the syslog message line. Only destination IP.
If I search for the "THREAT,url" patterrn, I get the destination URL and can do some sort of web usage reporting.
As I understand all webfiltering activity logs as "THREAT,url". However, it only llogs if the action is set to anything but "allow". When action is set to "allow", PA logs nothing at at all. Therefore we had to set all to "alert"
The TRAFFIC,start messages contain plenty of https (443) traffic, but do not log URLs. I was wandering if the TREAT message are included in the TRAFFIC messages (do they double up?) or the TRAFFIC messages include tflows that are not filtered for some reason. Below are the stats for a day. And the numbers just do not stack up.
4076679 TRAFFIC end
3468509 TRAFFIC start
1235392 THREAT url
483048 TRAFFIC drop
2103 TRAFFIC deny
517 THREAT vulnerability
5 THREAT spyware
So the URL logs are a completely seperate thing; they aren't located in the Traffic or Threat logs unless it's being recorded because a threat or something of the like was identified.
You actually want to ensure that the firewall is configured to send the URL logs to your SIEM and that the URL Filtering profile assigned to outbound traffic is set to at least 'alert' on every single category so the URL actually gets recorded. You then would have to utilize an extractor to record the actual fields within the message to the proper tag and then build your report in correlation with the Traffic logs and the URL logs if you want a report similar to what your proxy was likely building.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!