topic Re: Re: How to increase amount of log data removed by database purge in General Topics

How to increase amount of log data removed by database purge

EdwinD — Mon, 03 Feb 2014 21:09:40 GMT

I currently have a very large number of "Current size of threat log database exceeds alarm threshold."

On occasion I do see that logging stops at some point during the day and then resumes after the nightly database purge occurs.

I would like for the nightly purge to purge out more of the log data than it is currently purging. I would like to not see this alarm as it could hide more important alarms.

How can I configure the nightly log database purge to truncate more of the log file? If I can't, can I run the purge more frequently than nightly?

Thank you.

Re: How to increase amount of log data removed by database purge

HULK — Tue, 04 Feb 2014 16:45:36 GMT

Re: How to increase amount of log data removed by database purge

EdwinD — Thu, 06 Feb 2014 20:00:20 GMT

Thank you.

Currently my URL filter has quit logging because (I assume) it is full.

>>The quota is checked each time a logdb file is rotated. If the quota threshold is violated then we start deleting logs starting from the oldest until the threshold is no longer exceeded. To see how often the logdb file is rotating, you can review the ms.log file for the following entry "Initing log file with version".

Can I manually run the logdb rotation? If so, how?

Re: How to increase amount of log data removed by database purge

Phoenix — Thu, 06 Feb 2014 21:35:51 GMT

Hello EdwinD

The mechanism of purging logs to clear space is an automatic process. It kicks in only after reaching a certain free space is left. Manual control we have is to clear the whole space at once this is even seen in Device tab to clear logs.

But to clear partially it is done by the PAN auto job internally or has to be done in the root.

Thanks

Re: How to increase amount of log data removed by database purge

EdwinD — Thu, 06 Feb 2014 21:47:28 GMT

Perhaps I'm looking at this wrong. When the URL filtering logs quit logging, what is the appropriate action to take to get URL logging again? I am assuming that the logs are full and thus I have no logs. Perhaps that is not the problem at all.

Re: How to increase amount of log data removed by database purge

Phoenix — Thu, 06 Feb 2014 23:02:21 GMT

Hello EdwinD

To find if the URL logs are not being written run the command

debug log-receiver statistics

This would give how many URL logs are written and we should see if this is increasing or stays at same value when command is run multiple times.

Also the below command would give more stats about how the URL logs are being generated and details

debug device-server dump dynamic-url statistics

Example:

Total URL processed: 423

Total URL received: 423

Total URL request loss: 0

Total URL request discard(full): 0

Total URL request discard(invalid): 0

Total URL request discard(alloc): 0

Total URL request allocation failure: 0

Url logs are being processed by the device server daemon, you can try to restart it to refresh its resources. This is management plane daemon and it should not affect the data plane traffic in general.

debug software restart device-server

Another command to see what is happening on the device through counters and specific to the url would be seen as below,

show counter global filter delta yes | match url

Still the points holds good where if there is no space for the device to store then it has to be cleared. We can verify this from command

show system logdb-quota

Hope this helps.

Re: Re: How to increase amount of log data removed by database purge

EdwinD — Fri, 07 Feb 2014 00:04:54 GMT

Thank you.

URL logs written: 10537835

Then a little later

URL logs written: 10560883

However, there are no URL logs for the last 6 hours in the web interface view.

The output from the other command leads me to believe we aren't dropping URL log requests.

Current read idx: 23604

Current write idx: 23604

Total URL processed: 89140

Total URL received: 89140

Total URL request loss: 0

Total URL request discard: 0

Current async read idx: 39227

Current async write idx: 39227

Total async URL processed: 39227

Total async URL received: 39227

Total async URL request discard: 0

debug software restart device-server - this did knock off all my Citrix Xenapp/Xendesktop users. I think this is because I have SSL decryption of this traffic enabled on the PA. This did not resolve the issue.

I'm not certain how to read the logdb-quota output. If I am reading it correctly, I am full because the threat logs exceed my 19GB qouta, and URL filtering goes to the threat logs.

Quotas:

traffic: 31.00%, 36.932 GB

threat: 16.00%, 19.061 GB

system: 4.00%, 4.765 GB

config: 4.00%, 4.765 GB

alarm: 3.00%, 3.574 GB

trsum: 7.00%, 8.339 GB

hourlytrsum: 3.00%, 3.574 GB

dailytrsum: 1.00%, 1.191 GB

weeklytrsum: 1.00%, 1.191 GB

thsum: 2.00%, 2.383 GB

hourlythsum: 1.00%, 1.191 GB

dailythsum: 1.00%, 1.191 GB

weeklythsum: 1.00%, 1.191 GB

appstat: 6.00%, 7.148 GB

userid: 1.00%, 1.191 GB

hipmatch: 3.00%, 3.574 GB

application-pcaps: 1.00%, 1.191 GB

threat-pcaps: 1.00%, 1.191 GB

debug-filter-pcaps: 1.00%, 1.191 GB

hip-reports: 1.00%, 1.191 GB

dlp-logs: 1.00%, 1.191 GB

Disk usage:

traffic: Logs: 15G, Index: 16G

threat: Logs: 9.3G, Index: 11G

system: Logs: 207M, Index: 43M

config: Logs: 1.3G, Index: 9.9M

alarm: Logs: 72K, Index: 36K

trsum: Logs: 4.2G, Index: 4.3G

hourlytrsum: Logs: 3.1G, Index: 412M

dailytrsum: Logs: 1.1G, Index: 147M

weeklytrsum: Logs: 1.1G, Index: 148M

thsum: Logs: 2.4G, Index: 79M

hourlythsum: Logs: 56M, Index: 42M

dailythsum: Logs: 34M, Index: 40M

weeklythsum: Logs: 24M, Index: 9.0M

appstatdb: Logs: 288M, Index: 125M

userid: Logs: 1.2G, Index: 440K

hipmatch: Logs: 16K, Index: 16K

application-pcaps: 681M

threat-pcaps: 4.0K

debug-filter-pcaps: 8.0K

dlp-logs: 4.0K

hip-reports: 1.1M

wildfire: 4.0K

show log threat - this shows that I have logs going back to 12/11/2013.

I lowered the threat DB from 16% to 7% and then did a commit. After the commit, an Exec task ran for some time. After the Exec task completed, I started receiving URL filtering logs again. I believe reducing the threat DB caused the database purge to occur.

I increased the threat DB back to 16% and did another commit. This seems to be a work around for this issue.

Re: How to increase amount of log data removed by database purge

HULK — Fri, 07 Feb 2014 00:27:52 GMT

It's looking like the log-receiver process was restarted during the commit and started populating logs again. As an addition advice, whenever you will face such an issue, please take a look at the devsrvr ( device server) , logrcvr ( Log receiver), mgmtsrvr ( management server) CPU and memory utilization etc.

Thanks