Log Retention ... what, how, where?

by ‎02-09-2017 01:21 AM - edited ‎05-11-2017 08:55 AM (1,320 Views)

Ever returning questions on Log Retention.  Quite a popular topic as seen in the screenshot below.

 

Log RetentionLog Retention

 

 

It's actually very simple.  It all depends on how much you are logging in combination with how much space you have available.  If you are logging EVERYTHING and keep all your logs locally on your firewall then it's not uncommon that you'll only have a couple of days of logs available... possibly even less.

 

If you have a M-100 or M-500 Panorama then you can add more hard drives.  If it is a virtual Panorama, you can allocate more log storage.  In doing so you can extend your log retention.

 

When you run out of space, the Palo Alto Firewall will automaticaly delete the oldest entries in that specific log.  When you are limited to store your logs locally then you can adjust the reserved space for each type of log in Device -> Setup -> Management -> Logging and Reporting Settings as seen in the screenshot below

 

Logging and Reporting SettingsLogging and Reporting Settings

 

If you see a drastic change in log retention then the most likely reason would be that there's currently more traffic hitting a rule that is being logged.  To retain the same log retention you would need to adjust your storage allocation.  That being said, it might also be a good idea to review what you are logging exactly.  Is it really necessary to log at start AND end of a session ? Do you really need all those internal (trusted) sessions logged ? 

 

In the newer PAN-OS versions there's a cool feature in ACC which allows you to see how many times a specific rule was hit over time.  This makes it easier to troubleshoot a sudden decrease in log retention while searching for that rule which is eating up all your available log space :

 

Rule UsageRule Usage

 

I'm by no means a legal expert but certain kind of security logs need a much longer retention than just a couple of days.  With that in mind it might even be a good idea to look into alternative log storage solutions when you are planning for long term log retention.  This is especially true when you have a very high log rate.  Your local device will not be able to hold your logs for that long but a M-100 or M-500 Panorama or a log collector might be the solution you need.

 

So how can you check your current log retention ?

Very simply using the CLI command 'show system logdb-quota'.  It will spit out the current retention for each type of log file on your local firewall :

 

>show system logdb-quota

....

Disk usage: traffic: Logs and Indexes: 26G Current Retention: 340 days threat: Logs and Indexes: 3.0G Current Retention: 829 days system: Logs and Indexes: 1.2G Current Retention: 731 days config: Logs and Indexes: 1.2G Current Retention: 1007 days trsum: Logs and Indexes: 3.7G Current Retention: 998 days hourlytrsum: Logs and Indexes: 1.2G Current Retention: 998 days dailytrsum: Logs and Indexes: 160M Current Retention: 998 days weeklytrsum: Logs and Indexes: 76M Current Retention: 996 days thsum: Logs and Indexes: 850M Current Retention: 998 days hourlythsum: Logs and Indexes: 66M Current Retention: 998 days dailythsum: Logs and Indexes: 21M Current Retention: 998 days weeklythsum: Logs and Indexes: 7.2M Current Retention: 996 days appstatdb: Logs and Indexes: 82M Current Retention: 1007 days userid: Logs and Indexes: 46M Current Retention: 976 days hipmatch: Logs and Indexes: 132K Current Retention: 916 days extpcap: Logs and Indexes: 20K Current Retention: 0 days urlsum: Logs and Indexes: 580M Current Retention: 342 days hourlyurlsum: Logs and Indexes: 306M Current Retention: 342 days dailyurlsum: Logs and Indexes: 26M Current Retention: 342 days weeklyurlsum: Logs and Indexes: 9.1M Current Retention: 282 days application: Logs and Indexes: 39M Current Retention: 959 days filters: Logs and Indexes: 28M Current Retention: 621 days dlp: Logs and Indexes: 20K Current Retention: 175 days hip_report_base: Logs and Indexes: 1.6M Current Retention: N/A wildfire: Logs and Indexes: 40K Current Retention: N/A

 

Please post questions you would like to have answered, comments or suggestions below !

 

But before doing that, you might want to check on our Live Community.  

 

Your question might already have been answered.  Below is just a small set of log retention related articles on the Live Community :

 

https://live.paloaltonetworks.com/t5/General-Topics/Logs-Retention/m-p/41106#M30203

https://live.paloaltonetworks.com/t5/General-Topics/Log-Retention/m-p/142139#M48497

https://live.paloaltonetworks.com/t5/General-Topics/Retention-period-for-traffic-logs-on-Panorama/m-...

https://live.paloaltonetworks.com/t5/Management-Articles/Critical-Panorama-Alarm-Minimum-Retention-P...

https://live.paloaltonetworks.com/t5/General-Topics/PA-3020-log-retention-period/m-p/134776#M47389

https://live.paloaltonetworks.com/t5/SME-Discussions/Threat-log-retention-is-very-short/m-p/132336#M...

 

 

Cheers !

@kiwi

 

Comments
by rjdahav163
yesterday

 

Hi

we are using Panorama v7.1.10  to manae our firewalls and logs are forwarded to panorama (except our 7050)

Now we set a expiration period of 60 days in panorama under:

Panorama > Setup > Management and edit the Logging and Reporting Settings AND

Panorama > Collector Groups

 

However I still see logs older than 60 days.

 

 

What we see is that the retention period is set correctly on the individual firewalls managed by Panorama.

However on the Panorama itself,  I see retention periods way older 60 days (eg:300 days) and thus the logs also from 300 days ago.

 

How do I correct that on Panorama? 

Ask Questions Get Answers Join the Live Community