A Debug Command to Clean Logs Automatically

[reaper]

There's a debug command that can help you clean up old logs automatically

Several of our customers have reported in the past that their systems were having trouble with available disk space on the management plane.

In most cases, it turned out that management process logs had become overweight and filled up more disk space than desired. This is because log records are not simply purged when the log file grows large, but an 'archive' is created that stores older logs up to a total of 4 additional versions. This is the expected behavior: if debugging is enabled on one or more of the management plane processes (device server, management server, ..), this will temporarily cause additional logs to be written and the log to grow in size more rapidly. Recent history is not immediately purged out, and some history can be retained before losing this information for future reference or troubleshooting purposes by creating an 'old' log and starting a fresh log.

admin@PA-5220> ls long-format yes mp-log mp-monitor*
-rw-r--r-- 1 root root   455144 Jul  3 05:45 /var/log/pan/mp-monitor.log
-rw-r--r-- 1 root root 10481820 Jul  3 04:58 /var/log/pan/mp-monitor.log.1
-rw-r--r-- 1 root root 10485513 Jul  2 09:54 /var/log/pan/mp-monitor.log.2
-rw-r--r-- 1 root root 10485393 Jul  1 14:54 /var/log/pan/mp-monitor.log.3
-rw-r--r-- 1 root root 10485585 Jun 30 19:50 /var/log/pan/mp-monitor.log.4

As you can see from the output above, some processes can be chatty in their logs and can retain several 'old' files so history is preserved for longer than a (few) day(s).

If several processes need the extra space at the same time, however, disk space may become scarce. An administrator can go in and delete older log files manually, but in case this task is cumbersome, frequent, and/or log retention is not crucial, a debug command has been introduced in PAN-OS 8.0.7 as PAN-79671 that can be set to automatically purge all 'old' logs when disk capacity reaches 95% of full:

debug software disk-usage aggressive-cleaning enable 
debug software disk-usage aggressive-cleaning disable 

When aggressive-cleaning is enabled, the system will not interfere with 'old' log files for as long as the disk capacity is below 95%. Once the high mark is reached, the system will automatically purge all the old (*.log.old , *.log.{1..4} ) files on the management plane to make room.

When the debug command is disabled, (default setting) the system will only purge any files that would go above *.log.4,

eg. *.log.4 is purged, *.log3 is renamed to *.log.4, *.log.2 is renamed to *.log.3 and so on, and a fresh *.log is started.

The debug is visible from the system state, once enabled.

admin@PA-5220> debug software disk-usage aggressive-cleaning enable 
This will automatically purge all old log files if disk hits 95% occupancy. Do you accept this potential loss of debuggability? (y or n) 

admin@PA-5220> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

Stay frosty,

Reaper