syslog reports (web usage)

[au_igs]

Hi all,

I do daily scripted syslog reports for traffic through the firewall. PA syslog messages are pretty good actually. However, the only messages that have something that resembles URL is the messages of the "THREAT url" pattern. Now, I accasionally need to do a web usage report for some managers on what their employees are doing. For that I need full URLs for the session along with more information, eg user agent, type of HTML request (get/post) etc. Not just for THREAT pattern, but for the normal traffic pattern (TRAFFIC start/end) as well. PA capable of that at all?

TRAFFIC end
TRAFFIC start
THREAT url
TRAFFIC drop
TRAFFIC deny
THREAT vulnerability
THREAT spyware

[reaper]

you'll need to use an off-device tool (like a SIEM) to combine these logs as they are not part of the same log database, if you rely on automation if this only needs to happen every once in a while you could consider, rather than deploying a SIEM, to access the 'Unified' log view, filter the information you need and export that data as a CSV

[BPry]

@au_igs ,

I would recommend installing a SIEM anyways if you don't already have one if you are interacting with Syslog at all already. Graylog (Free/Paid) or Splunk (Paid) would be my go-to for something like this. 

Your request isn't entirely clear but does your firewall actually log the URL information for all of your traffic to begin with, or not? In most environments you wouldn't actually be setup to log every visited URL from your firewall, which would diminish the effectfullness of the requested report. You'd want to look at your URL logs on the firewall and ensure you are actually logging the accessed URL for the traffic you wish to log, and possibly make some configuration changes to your URL FIltering profile if you are not seeing the logs you require recorded on the firewall. 

[au_igs]

Hi, thank you for your reply. I really appriciate the help.

I already have the syslog server and the siem configured and all is well in that regard. Syslog is recieveing fine and logs are really easy to read and very well structured.

The problem I'm having is the web usage reporting. Because now we no longer have the internet proxy (which used to log users' activity) I need to get that information from the syslog messages coming from the PA.

If I search for the "TRAFFIC,start" (or TRAFFIC,end) pattern, there is no destination URL in the syslog message line. Only destination IP.

If I search for the "THREAT,url" patterrn, I get the destination URL and can do some sort of web usage reporting.

As I understand all webfiltering activity logs as "THREAT,url". However, it only llogs if the action is set to anything but "allow". When action is set to "allow", PA logs nothing at at all. Therefore we had to set all to "alert"

The TRAFFIC,start messages contain plenty of https (443) traffic, but do not log URLs. I was wandering if the TREAT message are included in the TRAFFIC messages (do they double up?) or the TRAFFIC messages include tflows that are not filtered for some reason. Below are the stats for a day. And the numbers just do not stack up.

4076679  TRAFFIC end
3468509  TRAFFIC start
1235392  THREAT url
 483048  TRAFFIC drop
   2103  TRAFFIC deny
    517  THREAT vulnerability
      5  THREAT spyware

[BPry]

@au_igs,

So the URL logs are a completely seperate thing; they aren't located in the Traffic or Threat logs unless it's being recorded because a threat or something of the like was identified.

You actually want to ensure that the firewall is configured to send the URL logs to your SIEM and that the URL Filtering profile assigned to outbound traffic is set to at least 'alert' on every single category so the URL actually gets recorded. You then would have to utilize an extractor to record the actual fields within the message to the proper tag and then build your report in correlation with the Traffic logs and the URL logs if you want a report similar to what your proxy was likely building. 

[au_igs]

yes we did that. We set all categories to "alert" so we are recording URLs for all categories. When searching the syslog file for the "inside,outside" and "THREAT,url" I get the URLs in field 32 fine. Extracting it and matching to source IP is really easy.

however, there is a lot of traffic going through the firewall with the "inside,outside" and "TRAFFIC,start" pattern. And lots of it on 80 and 443. Logically all that traffic should match the URL Filtering profile, right? Does that mean we have overlapping rules that do not have URL Filtering profile applied to them?

From your comment I reason that all "inside,outside" should match "THREAT,url" and there should not be "TRAFFIC,start" (or end) Am I missing correct? logically. Or am I missing somthing.

[BPry]

@au_igs 

You appear to be looking at just two of the log databases when you filter your syslog messages, the Threat and Traffic databases. 

There is a completely separate set of logs on the firewall in the URL database. This is where all of the URLs visited are actually located. 

[au_igs]

I would think so too. Is there a way to pass this information onto the syslog server?

[au_igs]

finally after a lot of searching I found it. URL logs are, in fact, exported as part of the THREAT string. Only they are reported as THREAT,url

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClqKCAS

it's really strange. I don't know why PA do it that way, but it is what it is