PA-3020 log retention period

[Ernest_James]

Hi Experts,

I am quite new to Palo Alto and I have some queries regarding the URL filter log retention, before we can generate user activty reports for browsed URLs for more than two weeks old, but now we can only see URL filter logs up to no more than 4 days.

What affects the log retention period and how can we generate a month old User Activity report for a specific user if logs are not present anymore.

[santonic]

Log retention is affected only by space on disk. When you run out of it PA automaticaly deletes oldest entries in that specific log, whether it's traffic, threat, URL...

You can adjust the reserved space for each type of log in Device -> Setup -> Management tab -> Logging and Reporting Settings

Within the limits of your hard drive capacity of course.

[Ernest_James]

Hi Santonic, thanks for the response.

So does this means that we suddenly have an huge amount of increase in traffic that cause the retention from more than 2 weeks to just 4days?

Also which one of this affects the url filter.

> show system logdb-quota

Quotas:
system: 4.00%, 3.356 GB
config: 4.00%, 3.356 GB
alarm: 3.00%, 2.517 GB
appstat: 6.00%, 5.034 GB
hip-reports: 1.00%, 0.839 GB
traffic: 32.00%, 26.850 GB
threat: 16.00%, 13.425 GB
trsum: 7.00%, 5.873 GB
hourlytrsum: 3.00%, 2.517 GB
dailytrsum: 1.00%, 0.839 GB
weeklytrsum: 1.00%, 0.839 GB
thsum: 2.00%, 1.678 GB
hourlythsum: 1.00%, 0.839 GB
dailythsum: 1.00%, 0.839 GB
weeklythsum: 1.00%, 0.839 GB
userid: 1.00%, 0.839 GB
application-pcaps: 1.00%, 0.839 GB
extpcap: 1.00%, 0.839 GB
debug-filter-pcaps: 1.00%, 0.839 GB
dlp-logs: 1.00%, 0.839 GB
hipmatch: 3.00%, 2.517 GB

[santonic]

Hmm, good question. All URL related log files seem to be 'summary' type.

[BPry]

Generally yes, if you see a drastic decrease in log retention then the only reason would be that you are seeing more traffic and would need to adjust your storage allocation if you want to retain more. That being said it's also probably a good idea to take a look and see if anybody changed/created a rule that is constatntly being logged or if they created something small that logs on start and end. I've run into that issue before where someone enables logging at start and end for testing but forgets to disable it and set the logging to end like we do on everything else. 

The URL filtering is part of the traffic report quota if memory serves correctly. 

[Ernest_James]

Hi BPry,

I am seeing two suspected rules with log at start, that is unusual form the rest which only logs at the end.

Now how can I prove that these are the guilty rules?

Are there any way to check how much they are logging? This is so that I can raise a change request for removing the logging at the start.

[BPry]

When you look at your traffic logs you can add the 'rule' column which will display the rule that was used and logged the action. As far as logging only at those two logs the best way would be to create a custom report with a rule eq 'whatever' statement to just get the logs for the two rules that you suspect. If it's logging at both start and end you will see many pages of results. 

Keep in mind that sometimes there is a legitimate reason that you would want to log at both start and end, but sometimes different admins will accidentally set it to both. 

[RFalconer]

One other thing you can check is the 'Max Rows in User Activity Report'. If you hit the maximum number of rows for the report based on 4 days of activity, it won't show any activity further back.  If you've changed the activity report to included detailed browsing, increasing the number of rows in the report, this would possibly cause an issue.

[Ernest_James]

Hi RFalconer,

As for the rows in the report, it was initially set to 50K but we are getting around 2 weeks worth or user activity logs, it was increased to the maximum value and still we are just getting around or less than 4days worth of user activity logs.

[santonic]

There is a pre-defined report (Reports->Traffic Reports->Security Rules) which will show you most used rules. Check if some irrelevant traffic is being logged (DNS, ICMP...) and if some of the most used rules log session start as well.

[kiwi]

Hi @Ernest_James,

The ACC also offers the information on 'Rule Usage' :

Rule Usage

Cheers !

-Kim.

[Brandon_Wertz]

@Ernest_James Traffic which matches your policy will definitely affect your device.  If possible you might want to modify what you log and when as far as URL logs.

For one function my company uses a 3020 pair and we've got logs back before the 20th.  So if you've got a specific requirement it might be worth reallocating storage capacity from one log type to another.

[Ernest_James]

@kiwi

I do not see rule usage on my ACC, maybe im using a different version.

[Ernest_James]

@Brandon_Wertz

Quotas:
system:                     4.00%, 3.356 GB
config:                      4.00%, 3.356 GB
alarm:                       3.00%, 2.517 GB
appstat:                   6.00%, 5.034 GB
hip-reports:            1.00%, 0.839 GB
traffic:                    32.00%, 26.850 GB
threat:                    16.00%, 13.425 GB
trsum:                      7.00%, 5.873 GB
hourlytrsum:           3.00%, 2.517 GB
dailytrsum:              1.00%, 0.839 GB
weeklytrsum:          1.00%, 0.839 GB
thsum:                     2.00%, 1.678 GB
hourlythsum:          1.00%, 0.839 GB
dailythsum:             1.00%, 0.839 GB
weeklythsum:         1.00%, 0.839 GB
userid:                     1.00%, 0.839 GB
application-pcaps: 1.00%, 0.839 GB
extpcap:                  1.00%, 0.839 GB
debug-filter-pcaps: 1.00%, 0.839 GB
dlp-logs:                   1.00%, 0.839 GB
hipmatch:                 3.00%, 2.517 GB

[Ernest_James]

@santonic

I have checked the Reports>Traffic Reports>Security Rules and found out this:

Site A has log problems with 4 days worth of user activity logs, Site B which has 30G less than SiteA, can hold up to 3 months of user activity logs.

Please correct me if I am wrong, but Monitor>PDF Reports>User Avtivity Report should be basically text file logs arranged into PDF for better viewing, right? In my opinion, it should not take a lot of space to retain this logs.