topic Re: PA-3020 log retention period in General Topics

PA-3020 log retention period

Ernest_James — Wed, 28 Dec 2016 06:59:18 GMT

Hi Experts,

I am quite new to Palo Alto and I have some queries regarding the URL filter log retention, before we can generate user activty reports for browsed URLs for more than two weeks old, but now we can only see URL filter logs up to no more than 4 days.

What affects the log retention period and how can we generate a month old User Activity report for a specific user if logs are not present anymore.

Re: PA-3020 log retention period

santonic — Wed, 28 Dec 2016 07:20:18 GMT

Log retention is affected only by space on disk. When you run out of it PA automaticaly deletes oldest entries in that specific log, whether it's traffic, threat, URL...

You can adjust the reserved space for each type of log in Device -> Setup -> Management tab -> Logging and Reporting Settings

Within the limits of your hard drive capacity of course.

Re: PA-3020 log retention period

Ernest_James — Wed, 28 Dec 2016 07:30:59 GMT

Hi Santonic, thanks for the response.

So does this means that we suddenly have an huge amount of increase in traffic that cause the retention from more than 2 weeks to just 4days?

Also which one of this affects the url filter.

> show system logdb-quota

Quotas:
system: 4.00%, 3.356 GB
config: 4.00%, 3.356 GB
alarm: 3.00%, 2.517 GB
appstat: 6.00%, 5.034 GB
hip-reports: 1.00%, 0.839 GB
traffic: 32.00%, 26.850 GB
threat: 16.00%, 13.425 GB
trsum: 7.00%, 5.873 GB
hourlytrsum: 3.00%, 2.517 GB
dailytrsum: 1.00%, 0.839 GB
weeklytrsum: 1.00%, 0.839 GB
thsum: 2.00%, 1.678 GB
hourlythsum: 1.00%, 0.839 GB
dailythsum: 1.00%, 0.839 GB
weeklythsum: 1.00%, 0.839 GB
userid: 1.00%, 0.839 GB
application-pcaps: 1.00%, 0.839 GB
extpcap: 1.00%, 0.839 GB
debug-filter-pcaps: 1.00%, 0.839 GB
dlp-logs: 1.00%, 0.839 GB
hipmatch: 3.00%, 2.517 GB

Re: PA-3020 log retention period

santonic — Wed, 28 Dec 2016 07:42:40 GMT

Hmm, good question. All URL related log files seem to be 'summary' type.

Re: PA-3020 log retention period

BPry — Wed, 28 Dec 2016 14:40:07 GMT

Generally yes, if you see a drastic decrease in log retention then the only reason would be that you are seeing more traffic and would need to adjust your storage allocation if you want to retain more. That being said it's also probably a good idea to take a look and see if anybody changed/created a rule that is constatntly being logged or if they created something small that logs on start and end. I've run into that issue before where someone enables logging at start and end for testing but forgets to disable it and set the logging to end like we do on everything else.

The URL filtering is part of the traffic report quota if memory serves correctly.

Re: PA-3020 log retention period

Ernest_James — Wed, 28 Dec 2016 15:10:09 GMT

Hi BPry,

I am seeing two suspected rules with log at start, that is unusual form the rest which only logs at the end.

Now how can I prove that these are the guilty rules?

Are there any way to check how much they are logging? This is so that I can raise a change request for removing the logging at the start.

Re: PA-3020 log retention period

BPry — Wed, 28 Dec 2016 15:42:12 GMT

When you look at your traffic logs you can add the 'rule' column which will display the rule that was used and logged the action. As far as logging only at those two logs the best way would be to create a custom report with a rule eq 'whatever' statement to just get the logs for the two rules that you suspect. If it's logging at both start and end you will see many pages of results.

Keep in mind that sometimes there is a legitimate reason that you would want to log at both start and end, but sometimes different admins will accidentally set it to both.

Re: PA-3020 log retention period

RFalconer — Wed, 28 Dec 2016 22:14:24 GMT

One other thing you can check is the 'Max Rows in User Activity Report'. If you hit the maximum number of rows for the report based on 4 days of activity, it won't show any activity further back. If you've changed the activity report to included detailed browsing, increasing the number of rows in the report, this would possibly cause an issue.

Re: PA-3020 log retention period

Ernest_James — Thu, 29 Dec 2016 01:28:22 GMT

Hi RFalconer,

As for the rows in the report, it was initially set to 50K but we are getting around 2 weeks worth or user activity logs, it was increased to the maximum value and still we are just getting around or less than 4days worth of user activity logs.

Re: PA-3020 log retention period

santonic — Thu, 29 Dec 2016 06:48:12 GMT

There is a pre-defined report (Reports->Traffic Reports->Security Rules) which will show you most used rules. Check if some irrelevant traffic is being logged (DNS, ICMP...) and if some of the most used rules log session start as well.

Re: PA-3020 log retention period

kiwi — Thu, 29 Dec 2016 09:17:01 GMT

Hi @Ernest_James,

The ACC also offers the information on 'Rule Usage' :

Rule Usage

Cheers !

-Kim.

Re: PA-3020 log retention period

Brandon_Wertz — Thu, 29 Dec 2016 22:35:07 GMT

@Ernest_James Traffic which matches your policy will definitely affect your device. If possible you might want to modify what you log and when as far as URL logs.

For one function my company uses a 3020 pair and we've got logs back before the 20th. So if you've got a specific requirement it might be worth reallocating storage capacity from one log type to another.

Re: PA-3020 log retention period

Ernest_James — Fri, 30 Dec 2016 01:37:12 GMT

@kiwi

I do not see rule usage on my ACC, maybe im using a different version.

Re: PA-3020 log retention period

Ernest_James — Fri, 30 Dec 2016 01:42:25 GMT

@Brandon_Wertz

Re: PA-3020 log retention period

Ernest_James — Fri, 30 Dec 2016 02:12:07 GMT

@santonic

I have checked the Reports>Traffic Reports>Security Rules and found out this:

Site A has log problems with 4 days worth of user activity logs, Site B which has 30G less than SiteA, can hold up to 3 months of user activity logs.

Please correct me if I am wrong, but Monitor>PDF Reports>User Avtivity Report should be basically text file logs arranged into PDF for better viewing, right? In my opinion, it should not take a lot of space to retain this logs.

Re: PA-3020 log retention period

santonic — Fri, 30 Dec 2016 07:22:04 GMT

Transfered bytes are irrelevant for logging. Log entries are generated per session so look at seesions counter values. A single http download session which transfer 3Gb means one log entry same as a DNS query for this site which transfers only few bytes.

Check the most used rules and see if you log some non relevant sessions like DNS and ICMP or boradcast traffic and similar.

Re: PA-3020 log retention period

santonic — Fri, 30 Dec 2016 07:34:52 GMT

Reports are basicaly queries on log files for specific information. So they are sort of an extract of log files. And I believe they are stored seperately from log files so they don't affect log retention directly.

Re: PA-3020 log retention period

kiwi — Fri, 30 Dec 2016 07:52:17 GMT

Hi @Ernest_James,

That's possible. ACC got a major facelift in PAN-OS 7.0 and some features were added. Possibly pre-7.0 won't have it.

It will basically return the same output as seen in the Reports>Traffic Reports>Security Rules. As santonic already pointed out you need to check the number of sessions.

Cheers !

-Kim.

Re: PA-3020 log retention period

Ernest_James — Wed, 04 Jan 2017 11:25:03 GMT

@kiwi @santonic

I have checked the most used rule but it has been there before. as for the rule with log on start as well seem not to be that used much.

Any other suggestions?

Re: PA-3020 log retention period

santonic — Wed, 04 Jan 2017 12:21:04 GMT

Even if it's been there always you can optimise it and turn off logging for non interesting traffic.

But to find the source of spike of events: PA FW saves these reports daily. I guess you have to check past reports, find out on which day there was a spike, which rule recorded it and (in the unlikely case you still have logs for that day) you can find out which traffic caused it. If you don't have logs you can check other automated reports and look for possible causes.