A Debug Command to Clean Logs Automatically

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
Cyber Elite
Cyber Elite

There's a debug command that can help you clean up old logs automatically

 

Several of our customers have reported in the past that their systems were having trouble with available disk space on the management plane.

 

In most cases, it turned out that management process logs had become overweight and filled up more disk space than desired. This is because log records are not simply purged when the log file grows large, but an 'archive' is created that stores older logs up to a total of 4 additional versions. This is the expected behavior: if debugging is enabled on one or more of the management plane processes (device server, management server, ..), this will temporarily cause additional logs to be written and the log to grow in size more rapidly. Recent history is not immediately purged out, and some history can be retained before losing this information for future reference or troubleshooting purposes by creating an 'old' log and starting a fresh log.

 

admin@PA-5220> ls long-format yes mp-log mp-monitor*
-rw-r--r-- 1 root root   455144 Jul  3 05:45 /var/log/pan/mp-monitor.log
-rw-r--r-- 1 root root 10481820 Jul  3 04:58 /var/log/pan/mp-monitor.log.1
-rw-r--r-- 1 root root 10485513 Jul  2 09:54 /var/log/pan/mp-monitor.log.2
-rw-r--r-- 1 root root 10485393 Jul  1 14:54 /var/log/pan/mp-monitor.log.3
-rw-r--r-- 1 root root 10485585 Jun 30 19:50 /var/log/pan/mp-monitor.log.4

As you can see from the output above, some processes can be chatty in their logs and can retain several 'old' files so history is preserved for longer than a (few) day(s).

 

If several processes need the extra space at the same time, however, disk space may become scarce. An administrator can go in and delete older log files manually, but in case this task is cumbersome, frequent, and/or log retention is not crucial, a debug command has been introduced in PAN-OS 8.0.7 as PAN-79671 that can be set to automatically purge all 'old' logs when disk capacity reaches 95% of full:

 

debug software disk-usage aggressive-cleaning enable 
debug software disk-usage aggressive-cleaning disable 

When aggressive-cleaning is enabled, the system will not interfere with 'old' log files for as long as the disk capacity is below 95%. Once the high mark is reached, the system will automatically purge all the old (*.log.old , *.log.{1..4} ) files on the management plane to make room.

 

When the debug command is disabled, (default setting) the system will only purge any files that would go above *.log.4,

eg. *.log.4 is purged, *.log3 is renamed to *.log.4, *.log.2 is renamed to *.log.3 and so on, and a fresh *.log is started.

 

The debug is visible from the system state, once enabled.

 

admin@PA-5220> debug software disk-usage aggressive-cleaning enable 
This will automatically purge all old log files if disk hits 95% occupancy. Do you accept this potential loss of debuggability? (y or n) 

admin@PA-5220> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

 

 

 

Stay frosty,

Reaper

 

18 Comments
L2 Linker

When I run this I get the following:

 


Server error : Failed to execute op command

Cyber Elite
Cyber Elite

hi @dstjames

 

are you on 8.0.7 or later?

L2 Linker

Yeah 8.1.1. 

 

 

Cyber Elite
Cyber Elite

hi @dstjames

Have you tried restarting the management plane ( > request restart software ) ? May want to give that a try

If that doesn't help you may want to reach out to tac to have a look at what may be keeping you from exxecuting this command

 

Cyber Elite
Cyber Elite

like this post

very helpfull

L1 Bithead

I am on version 8.0.10 and running

show system state | match aggressive-clean

displays nothing. Which according to the article has the aggresive clean disabled. However, it does not seem to be deleting the log files mentioned and I would need to delete those files manually.

Is there a way to automate this task?

Cyber Elite
Cyber Elite

Hi @KatiaNunez 

 

This is expected behavior

If you enable the command, this will start the automated task

 

Cyber Elite
Cyber Elite

i have configured this command but still got email alert 

 

-NGFW-1(active)> show system state | match aggressive-cleaning

 

 

 

 


cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

domain: 1
receive_time: 2019/04/29 05:03:23
serial: 002201001803
seqno: 6880362
actionflags: 0x8000000000000000
type: SYSTEM
subtype: general
config_ver: 0
time_generated: 2019/04/29 05:03:23
dg_hier_level_1: 0
dg_hier_level_2: 0
dg_hier_level_3: 0
dg_hier_level_4: 0
vsys_name:
device_name: NGFW-1
vsys_id: 0
vsys:
eventid: general
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Disk usage for / exceeds limit, 96 percent in use, cleaning filesystem

 

I thought when you configure aggressive cleaning it should do this automaticalls and we should not get email alert?

Cyber Elite
Cyber Elite

The last line says it is cleaning

But it only cleans logs, there may be core files or something else thats taking up space

What platform is this? 

Cyber Elite
Cyber Elite

Here is required info

 

model: PA-5050
sw-version: 8.0.9

Cyber Elite
Cyber Elite

Have you checked if theres anything else, like > show system files

Cyber Elite
Cyber Elite

NGFW-1(active)> show system files

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

/opt/var.dp2/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp2/cores/crashinfo:
total 0

/opt/var.dp1/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp1/cores/crashinfo:
total 0

/opt/var.dp0/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp0/cores/crashinfo:
total 0

/opt/var.cp/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.cp/cores/crashinfo:
total 0

/opt/panlogs/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/panlogs/cores/crashinfo:
total 0

/var/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Mar 12 14:45 crashinfo

/var/cores/crashinfo:
total 0

 

 

NGFW-1(active)> show system disk-space

Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.8G 3.4G 242M 94% /
/dev/md5 7.6G 3.5G 3.8G 48% /opt/pancfg
/dev/md6 3.8G 2.8G 852M 77% /opt/panrepo
tmpfs 2.0G 116M 1.9G 6% /dev/shm
cgroup_root 2.0G 0 2.0G 0% /cgroup
/dev/md8 198G 142G 46G 76% /opt/panlogs
tmpfs 12M 0 12M 0% /opt/pancfg/mgmt/lcaas/ssl/private

Cyber Elite
Cyber Elite

@MP18,

So you could be running into PAN-96522 where your logs aren't rotating currectly, PAN-92958 where the firewall isn't archiving and rotating /var/on file.

8.0.9 is relatively old; I would recommend upgrading to something a bit more current in that branch, past 8.0.14 if you want to ensure you aren't running into 96522. 

Cyber Elite
Cyber Elite

i will do the upgrade on the change window.

can you please explain about below in more detail please

 

 PAN-92958 where the firewall isn't archiving and rotating /var/on file.

Cyber Elite
Cyber Elite

https://docs.paloaltonetworks.com/pan-os/8-0/pan-os-release-notes/pan-os-8-0-addressed-issues/pan-os...

 

 

Fixed an issue where disk utilization increased unnecessarily because the firewall did not archive and rotate the /var/on file, which therefore grew to over 40MB.p> 

Cyber Elite
Cyber Elite

MAny thanks Reaper 

L1 Bithead

So, the solution is to upgrade to a new version?

We have had this issue in every version we have updated, and after the update the disk issue comes back months later. Every time we open a ticket, Palo alto support tells that the solution is to upgrade to a new OS version. In our case updating to a new version will fix the issue just temporarily.

 

Cyber Elite
Cyber Elite

@KatiaNunez,

Please create a discussion on the 'General Topics' specfic to your issue. Include your model and software version along with the output of 'show system files'. 

  • 114035 Views
  • 18 comments
  • 3 Likes
Register or Sign-in
Labels