Logging - Best Practise?

[networkadmin]

What is considered "best practise" to get useful logfiles should the need arise to go through them?

The default seems to be CSV of source and destination IP/User but, for example, how would I get the URL visited since 99% of the time that is the bit we would be looking for, or would recognize more than an IP address?

I guess it could be done by using a URL profile with all categories that aren't blocked set to "alert", but would this be included in the traffic log that is exported via FTP?

Just after some guidance on the right/best way to be doing this.

Thanks.

[mjacobsen · Accepted solution]

There are two primary methods of archiving logs. The most common method is to setup log forwarding profiles to send the logs off the device in real-time via syslog. An alternative would be to schedule a nightly FTP of the logs if you do not have a syslog collector available.

As far as the URL information, this is only present in the URL filtering logs. A given session (represented by a traffic log) may have many URLs associated with it so logging the URL in the traffic log would not work. If the URL is the key info you are looking to archive then use one of the two methods above for getting those logs off of the device in an automated fashion.

Hope this helps.

Mike

[networkadmin]

Thanks for the reply Mike.

Real-time isn't a requirement (yet) so I think for now the nightly FTP export is sufficient.

Looking at the traffic logs that exported the last couple of nights, I don't see any URL info though?

Incidentally, is there any option that I've missed (or can a feature request be put in?) to compress the exported logfile as it's a few hundred mb of raw text per day.

[mjacobsen]

The traffic logs will not have any URL information. That will only be in the URL logs. You should be able to get this data by using the "ftp export log threat..." commands. This can also be schedule to happen on a nightly basis using the Scheduled Log Export option on the Device tab. (Currently, threat, URL and data filtering logs are all exported together under the "threat" tag. This will be broken out explicitly in PAN-OS 3.1.)

Mike

[networkadmin]

Thanks, so for now the solution is:

Enable "alert" for all URL categories (with block and other exceptions as applicable) in the main outbound policy.

Enable export of "threat" logs to FTP (or syslog if required).

Any thoughts on my query about compressing logs as they're pretty darned big...

[mjacobsen]

When doing an SCP export, we do zip the logs in a stream based fashion without needing extra disk space for the compressed file before sending it to the destination. FTP doesn't allow for this by default, but we can look into a mechanism for this are into allow SCP to be scheduled.

Mike

[networkadmin]

Hmm OK well I just looked at my exported traffic and threat logs and no sign of any blocked URLs?

I can open a ticket with support but any suggestions?

[keklund]

I just tried this and am noticing the same thing (no URLs in threat export).

I do get the URL data in the syslogs, but that takes a few extra steps I'd like to avoid when collecting the data into SQL.

I need a good way to collect the URL data off the filter itself.

[networkadmin]

Bumpity bump.. Any way of logging this info off-box without Syslog until PAN OS 3.1 (I'm assuming this does it to FTP?)

[rps]

Here is another one that would be interrested in "gzip -9" of the url-logs into ftp-server as a daily (nightly) schedule 🙂