topic Re: Traffic log database exceeds alarm threshold in General Topics

Traffic log database exceeds alarm threshold

BSadozai — Wed, 10 Oct 2012 15:13:50 GMT

Hello,

This is not very clear on Palo box, since months we have issue that every
week we have alarm indicating that the log was exceeded 80 of the quota, in
fact we want to log all traffics and don’t want to disable logging on some
rules, I monitored during the week the logdb-quota and Palo don’t clear/purge
or delete older log at 80%, we opened a case on Palo support from three weeks
and the only response we get is that we have to disable some logging on our
rules.

So will be grate that Palo clarify this issue and response the question
below:

•Is Palo box purge/clear or delete older logs or it
overwrite older logs ?
•If it purge/clear
or delete, how many volume or % ?

•If it purge/clear

or delete, we should find a log entry on the system log and what is the exact event for that?
•Is there a specific time that Palo box do this or it do
it as soon as 80% was reached ?
•Is there a way to configure Palo box to clear or
deleted older logs from specific time ? if not way ( this is basic
configuration of system that have to be available)
•How can we sort out this issue? Without disabling logging
on some rules?

Thanks for your answer and help

Re: Traffic log database exceeds alarm threshold

mbutt — Wed, 10 Oct 2012 17:48:13 GMT

Hi,

Please see below and see if that helps answer your questions.

The logs are purged when the quota size is exhausted. This is why it has been recommended to set the overall quota to ~90% of the full disk. You do not have to save space, but it is recommended to improve performance.
The logs are purged to keep the log file as close to full as possible. If a partition is set to 100MB, the logs are not purged until the log file is 100% full (100MB+). The usage can be over the quota because the indexing will take up space, but it does not use the purging mechanism as the normal log writes. If the index takes place, but no new logs have come in, the usage can be over the quota (over 100MB for example) until the next log is written. Once the next log is written, the system will purge enough logs and index files to get below the quota.
If the amount of traffic logged is greater than what the firewall can delete, this alarm will be generated as explained in the above.
For deleting the logs partially currently there is no command. you can delete entire logs for example if you go to GUI and Device and manage logs it give you option to delete different logs.

If you are looking for partially deleting the logs the work around would be If you resize the partition, and commit and then size it back after the commit, you should essentially remove the last the oldest logs. For example if you have 1 GB of traffic logs, resize the partition to 500MB, and commit, you will remove the oldest 500MB of logs.

5. Also if you would like to request a feature to delete logs partially please contact your local SE (Sales Engineer) and he should be able to file an enhancement request for you.

Hope this helps.

Thanks

Numan

Re: Traffic log database exceeds alarm threshold

BSadozai — Mon, 15 Oct 2012 07:58:57 GMT

Hi,

Thanks for your answer but our device doesn’t purge logs
when quota size for traffic log is exhausted:

**********5.10.2012 - 16:00

Quotas:
traffic: 40.00%, 47.654 GB

traffic: Logs: 12G, Index: 14G

**********08.10.2012 - 10:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 14G, Index: 17G
threat: Logs: 524M, Index: 452M
system: Logs: 5.3M, Index: 3.2M
config: Logs: 167M, Index: 1.4M
alarm: Logs: 116K, Index: 68K
trsum: Logs: 1.9G, Index: 1.9G
hourlytrsum: Logs: 2.9G, Index: 420M
dailytrsum: Logs: 1.1G, Index: 160M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 199M, Index: 43M
hourlythsum: Logs: 112M, Index: 21M
dailythsum: Logs: 36M, Index: 8.9M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 28M, Index: 12M
userid: Logs: 655M, Index: 316K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 149M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********09.10.2012 - 10:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 15G, Index: 19G
threat: Logs: 543M, Index: 470M
system: Logs: 5.4M, Index: 3.2M
config: Logs: 175M, Index: 1.4M
alarm: Logs: 116K, Index: 68K
trsum: Logs: 1.9G, Index: 2.0G
hourlytrsum: Logs: 2.9G, Index: 423M
dailytrsum: Logs: 1.1G, Index: 160M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 204M, Index: 44M
hourlythsum: Logs: 117M, Index: 22M
dailythsum: Logs: 38M, Index: 9.3M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 29M, Index: 13M
userid: Logs: 675M, Index: 320K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 151M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********10.10.2012 - 16:00
Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 17G, Index: 21G
threat: Logs: 565M, Index: 498M
system: Logs: 5.5M, Index: 3.3M
config: Logs: 179M, Index: 1.5M
alarm: Logs: 116K, Index: 68K
trsum: Logs: 2.0G, Index: 2.0G
hourlytrsum: Logs: 3.1G, Index: 417M
dailytrsum: Logs: 1.1G, Index: 160M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 209M, Index: 46M
hourlythsum: Logs: 124M, Index: 23M
dailythsum: Logs: 40M, Index: 9.6M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 30M, Index: 13M
userid: Logs: 696M, Index: 324K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 158M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********11.10.2012 - 09:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 18G, Index: 22G
threat: Logs: 577M, Index: 509M
system: Logs: 5.5M, Index: 3.3M
config: Logs: 179M, Index: 1.5M
alarm: Logs: 132K, Index: 76K
trsum: Logs: 2.1G, Index: 2.1G
hourlytrsum: Logs: 2.9G, Index: 419M
dailytrsum: Logs: 1.1G, Index: 160M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 214M, Index: 47M
hourlythsum: Logs: 126M, Index: 23M
dailythsum: Logs: 41M, Index: 10M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 31M, Index: 14M
userid: Logs: 706M, Index: 328K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 163M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********12.10.2012 - 10:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 20G, Index: 24G
threat: Logs: 595M, Index: 527M
system: Logs: 5.6M, Index: 3.4M
config: Logs: 183M, Index: 1.5M
alarm: Logs: 168K, Index: 104K
trsum: Logs: 2.2G, Index: 2.1G
hourlytrsum: Logs: 2.8G, Index: 415M
dailytrsum: Logs: 1.1G, Index: 159M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 219M, Index: 49M
hourlythsum: Logs: 131M, Index: 24M
dailythsum: Logs: 43M, Index: 11M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 32M, Index: 14M
userid: Logs: 723M, Index: 332K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 169M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********12.10.2012 - 17:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 20G, Index: 25G
threat: Logs: 600M, Index: 536M
system: Logs: 5.6M, Index: 3.4M
config: Logs: 187M, Index: 1.5M
alarm: Logs: 172K, Index: 104K
trsum: Logs: 2.2G, Index: 2.2G
hourlytrsum: Logs: 3.0G, Index: 436M
dailytrsum: Logs: 1.1G, Index: 159M
weeklytrsum: Logs: 170M, Index: 26M
thsum: Logs: 219M, Index: 50M
hourlythsum: Logs: 133M, Index: 24M
dailythsum: Logs: 43M, Index: 11M
weeklythsum: Logs: 24M, Index: 4.6M
appstatdb: Logs: 32M, Index: 14M
userid: Logs: 730M, Index: 332K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 170M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

**********15.10.2012 - 9:00

Quotas:
             traffic: 40.00%, 47.654 GB
              threat: 8.00%, 9.531 GB
              system: 4.00%, 4.765 GB
              config: 4.00%, 4.765 GB
               alarm: 3.00%, 3.574 GB
               trsum: 7.00%, 8.339 GB
         hourlytrsum: 3.00%, 3.574 GB
          dailytrsum: 1.00%, 1.191 GB
         weeklytrsum: 1.00%, 1.191 GB
               thsum: 2.00%, 2.383 GB
         hourlythsum: 1.00%, 1.191 GB
          dailythsum: 1.00%, 1.191 GB
         weeklythsum: 1.00%, 1.191 GB
             appstat: 12.00%, 14.296 GB
              userid: 1.00%, 1.191 GB
            hipmatch: 3.00%, 3.574 GB
   application-pcaps: 1.00%, 1.191 GB
        threat-pcaps: 1.00%, 1.191 GB
debug-filter-pcaps: 1.00%, 1.191 GB
         hip-reports: 1.00%, 1.191 GB
            dlp-logs: 1.00%, 1.191 GB
Disk usage:
traffic: Logs: 22G, Index: 27G
threat: Logs: 639M, Index: 570M
system: Logs: 5.7M, Index: 3.5M
config: Logs: 187M, Index: 1.5M
alarm: Logs: 208K, Index: 116K
trsum: Logs: 2.3G, Index: 2.3G
hourlytrsum: Logs: 3.1G, Index: 439M
dailytrsum: Logs: 1.1G, Index: 159M
weeklytrsum: Logs: 212M, Index: 32M
thsum: Logs: 233M, Index: 52M
hourlythsum: Logs: 141M, Index: 26M
dailythsum: Logs: 46M, Index: 11M
weeklythsum: Logs: 30M, Index: 5.8M
appstatdb: Logs: 34M, Index: 15M
userid: Logs: 762M, Index: 344K
hipmatch: Logs: 16K, Index: 16K
application-pcaps: 183M
threat-pcaps: 4.0K
debug-filter-pcaps: 8.0K
dlp-logs: 4.0K
hip-reports: 1.1M
wildfire: 4.0K

So my question is how can I configure the box to clean for
example 20% of traffic logs when it reach 90 or 100%, it should be configurable
this is a basic configuration that have to be available?

Thanks

Re: Traffic log database exceeds alarm threshold

mbutt — Mon, 15 Oct 2012 16:55:47 GMT

Hi,

Do you have to manually go and delete the logs or are the new logs are still being written. If the new logs are still being written that means the logs are being purged. Also as mentioned the logs are purged to keep the log file as close to full as possible. If a partition is set to 100MB, the logs are not purged until the log file is 100% full (100MB+). The usage can be over the quota because the indexing will take up space, but it does not use the purging mechanism as the normal log writes. If the index takes place, but no new logs have come in, the usage can be over the quota (over 100MB for example) until the next log is written. Once the next log is written, the system will purge enough logs and index files to get below the quota.

Also if you would like to request a feature to delete logs partially please contact your local SE (Sales Engineer) and he should be able to file an enhancement request for you.

Thank you

Numan

Re: Traffic log database exceeds alarm threshold

BSadozai — Wed, 17 Oct 2012 12:16:37 GMT

Hello,

Thanks for your replay, so if I understand your explanations,
till the quota that I configured for the
traffic logs (traffic: 40.00%, 47.654 GB) is not reached by the
logs without index the device will not
purge logs ? as you can see below the traffic log on my device was reached at (traffic:
Logs: 22G, Index: 27G “total 49G” but logs only 22G) so what you mean is
that only when the logs part (in my case 22G) is reached to 47.654GB the device
will start purging ?

If this why there is alarm indicating that the current size
(49GB) of traffic log database exceeds alarm threshold value (90%) of total
allowed size (47.654 GB) ? and what I can do to not have this alarm ?

Thanks

Re: Traffic log database exceeds alarm threshold

Phoenix — Wed, 17 Oct 2012 13:04:47 GMT

Hi,

Seeing the latest comment I find that you do not want to see the alarms. You can clear the alarms.

Device Tab > Log Settings > Manage Logs > Clear Alarm Logs.

To view the logs before clearing you can see @ Monitor tab > Logs > Alarms.

The settings for the Alarms are done @ Device tab > Log setting > Alarms.

Here is an option to change thresholds or enable or disable alarms.

Hope this helps.

Re: Traffic log database exceeds alarm threshold

BSadozai — Wed, 17 Oct 2012 13:34:14 GMT

Hello,

Thanks for your comment, in fact I know all those settings,
but my question is about the purpose of this alarm, if I cannot configure the
log rotation I mean if I cannot setup (without asking to the Sales engineer
about a customized feature) the system to purge certain volume of logs when it
reach for example 90% of quota, I don’t see the purpose of this alarm.

And what happen on the device if I disable the alarm and the
quota is full every time ? its impact the performance of the devices ? I know as
explained above that the new logs will continue to write.

Thanks