syslog reports (web usage)

Showing results for 
Show  only  | Search instead for 
Did you mean: 

syslog reports (web usage)

L1 Bithead

Hi all,


I do daily scripted syslog reports for traffic through the firewall. PA syslog messages are pretty good actually. However, the only messages that have something that resembles URL is the messages of the "THREAT url" pattern. Now, I accasionally need to do a web usage report for some managers on what their employees are doing. For that I need full URLs for the session along with more information, eg user agent, type of HTML request (get/post) etc. Not just for THREAT pattern, but for the normal traffic pattern (TRAFFIC start/end) as well. PA capable of that at all?


THREAT vulnerability
THREAT spyware


Cyber Elite
Cyber Elite
you'll need to use an off-device tool (like a SIEM) to combine these logs as they are not part of the same log database, if you rely on automation if this only needs to happen every once in a while you could consider, rather than deploying a SIEM, to access the 'Unified' log view, filter the information you need and export that data as a CSV
Tom Piens
PANgurus - (co)managed services and consultancy

Cyber Elite
Cyber Elite

@au_igs ,

I would recommend installing a SIEM anyways if you don't already have one if you are interacting with Syslog at all already. Graylog (Free/Paid) or Splunk (Paid) would be my go-to for something like this. 

Your request isn't entirely clear but does your firewall actually log the URL information for all of your traffic to begin with, or not? In most environments you wouldn't actually be setup to log every visited URL from your firewall, which would diminish the effectfullness of the requested report. You'd want to look at your URL logs on the firewall and ensure you are actually logging the accessed URL for the traffic you wish to log, and possibly make some configuration changes to your URL FIltering profile if you are not seeing the logs you require recorded on the firewall. 

Hi, thank you for your reply. I really appriciate the help.


I already have the syslog server and the siem configured and all is well in that regard. Syslog is recieveing fine and logs are really easy to read and very well structured.


The problem I'm having is the web usage reporting. Because now we no longer have the internet proxy (which used to log users' activity) I need to get that information from the syslog messages coming from the PA.


If I search for the "TRAFFIC,start" (or TRAFFIC,end) pattern, there is no destination URL in the syslog message line. Only destination IP.


If I search for the "THREAT,url" patterrn, I get the destination URL and can do some sort of web usage reporting.


As I understand all webfiltering activity logs as "THREAT,url". However, it only llogs if the action is set to anything but "allow". When action is set to "allow", PA logs nothing at at all. Therefore we had to set all to "alert"


The TRAFFIC,start messages contain plenty of https (443) traffic, but do not log URLs. I was wandering if the TREAT message are included in the TRAFFIC messages (do they double up?) or the TRAFFIC messages include tflows that are not filtered for some reason. Below are the stats for a day. And the numbers just do not stack up.


4076679  TRAFFIC end
3468509  TRAFFIC start
1235392  THREAT url
 483048  TRAFFIC drop
   2103  TRAFFIC deny
    517  THREAT vulnerability
      5  THREAT spyware


So the URL logs are a completely seperate thing; they aren't located in the Traffic or Threat logs unless it's being recorded because a threat or something of the like was identified.

You actually want to ensure that the firewall is configured to send the URL logs to your SIEM and that the URL Filtering profile assigned to outbound traffic is set to at least 'alert' on every single category so the URL actually gets recorded. You then would have to utilize an extractor to record the actual fields within the message to the proper tag and then build your report in correlation with the Traffic logs and the URL logs if you want a report similar to what your proxy was likely building. 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!