There's a debug command that can help you clean up old logs automatically

Community Manager

Several of our customers have reported in the past that their systems were having trouble with available disk space on the management plane.

 

In most cases, it turned out that management process logs had become overweight and filled up more disk space than desired. This is because log records are not simply purged when the log file grows large, but an 'archive' is created that stores older logs up to a total of 4 additional versions. This is the expected behavior: if debugging is enabled on one or more of the management plane processes (device server, management server, ..), this will temporarily cause additional logs to be written and the log to grow in size more rapidly. Recent history is not immediately purged out, and some history can be retained before losing this information for future reference or troubleshooting purposes by creating an 'old' log and starting a fresh log.

 

admin@PA-5220> ls long-format yes mp-log mp-monitor*
-rw-r--r-- 1 root root   455144 Jul  3 05:45 /var/log/pan/mp-monitor.log
-rw-r--r-- 1 root root 10481820 Jul  3 04:58 /var/log/pan/mp-monitor.log.1
-rw-r--r-- 1 root root 10485513 Jul  2 09:54 /var/log/pan/mp-monitor.log.2
-rw-r--r-- 1 root root 10485393 Jul  1 14:54 /var/log/pan/mp-monitor.log.3
-rw-r--r-- 1 root root 10485585 Jun 30 19:50 /var/log/pan/mp-monitor.log.4

As you can see from the output above, some processes can be chatty in their logs and can retain several 'old' files so history is preserved for longer than a (few) day(s).

 

If several processes need the extra space at the same time, however, disk space may become scarce. An administrator can go in and delete older log files manually, but in case this task is cumbersome, frequent, and/or log retention is not crucial, a debug command has been introduced in PAN-OS 8.0.7 as PAN-79671  that can be set to automatically purge all 'old' logs when disk capacity reaches 95% of full:

 

debug software disk-usage aggressive-cleaning enable 
debug software disk-usage aggressive-cleaning disable 

When aggressive-cleaning is enabled, the system will not interfere with 'old' log files for as long as the disk capacity is below 95%. Once the high mark is reached, the system will automatically purge all the old (*.log.old , *.log.{1..4} ) files on the management plane to make room.

 

When the debug command is disabled, (default setting) the system will only purge any files that would go above *.log.4,

eg. *.log.4 is purged, *.log3 is renamed to *.log.4, *.log.2 is renamed to *.log.3 and so on, and a fresh *.log is started.

 

The debug is visible from the system state, once enabled.

 

admin@PA-5220> debug software disk-usage aggressive-cleaning enable 
This will automatically purge all old log files if disk hits 95% occupancy. Do you accept this potential loss of debuggability? (y or n) 

admin@PA-5220> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

 

 

 

Stay frosty,

Reaper

 

28,168 Views
Comments
L2 Linker

When I run this I get the following:

 


Server error : Failed to execute op command

27,925 Views
Community Manager

hi @dstjames

 

are you on 8.0.7 or later?

27,900 Views
L2 Linker

Yeah 8.1.1. 

 

 

26,691 Views
Community Manager

hi @dstjames

Have you tried restarting the management plane ( > request restart software ) ? May want to give that a try

If that doesn't help you may want to reach out to tac to have a look at what may be keeping you from exxecuting this command

 

26,679 Views
L4 Transporter

like this post

very helpfull

18,578 Views
L1 Bithead

I am on version 8.0.10 and running

show system state | match aggressive-clean

displays nothing. Which according to the article has the aggresive clean disabled. However, it does not seem to be deleting the log files mentioned and I would need to delete those files manually.

Is there a way to automate this task?

6,097 Views
Community Manager

Hi @KatiaNunez 

 

This is expected behavior

If you enable the command, this will start the automated task

 

6,086 Views
L4 Transporter

i have configured this command but still got email alert 

 

-NGFW-1(active)> show system state | match aggressive-cleaning

 

 

 

 


cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

domain: 1
receive_time: 2019/04/29 05:03:23
serial: 002201001803
seqno: 6880362
actionflags: 0x8000000000000000
type: SYSTEM
subtype: general
config_ver: 0
time_generated: 2019/04/29 05:03:23
dg_hier_level_1: 0
dg_hier_level_2: 0
dg_hier_level_3: 0
dg_hier_level_4: 0
vsys_name:
device_name: NGFW-1
vsys_id: 0
vsys:
eventid: general
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Disk usage for / exceeds limit, 96 percent in use, cleaning filesystem

 

I thought when you configure aggressive cleaning it should do this automaticalls and we should not get email alert?

5,098 Views
Community Manager

The last line says it is cleaning

But it only cleans logs, there may be core files or something else thats taking up space

What platform is this? 

5,081 Views
L4 Transporter

Here is required info

 

model: PA-5050
sw-version: 8.0.9

5,079 Views
Community Manager

Have you checked if theres anything else, like > show system files

5,077 Views
L4 Transporter

NGFW-1(active)> show system files

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

/opt/var.dp2/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp2/cores/crashinfo:
total 0

/opt/var.dp1/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp1/cores/crashinfo:
total 0

/opt/var.dp0/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.dp0/cores/crashinfo:
total 0

/opt/var.cp/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/var.cp/cores/crashinfo:
total 0

/opt/panlogs/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Jul 1 2018 crashinfo

/opt/panlogs/cores/crashinfo:
total 0

/var/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Mar 12 14:45 crashinfo

/var/cores/crashinfo:
total 0

 

 

NGFW-1(active)> show system disk-space

Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.8G 3.4G 242M 94% /
/dev/md5 7.6G 3.5G 3.8G 48% /opt/pancfg
/dev/md6 3.8G 2.8G 852M 77% /opt/panrepo
tmpfs 2.0G 116M 1.9G 6% /dev/shm
cgroup_root 2.0G 0 2.0G 0% /cgroup
/dev/md8 198G 142G 46G 76% /opt/panlogs
tmpfs 12M 0 12M 0% /opt/pancfg/mgmt/lcaas/ssl/private

5,074 Views
L7 Applicator

@MP18,

So you could be running into PAN-96522 where your logs aren't rotating currectly, PAN-92958 where the firewall isn't archiving and rotating /var/on file.

8.0.9 is relatively old; I would recommend upgrading to something a bit more current in that branch, past 8.0.14 if you want to ensure you aren't running into 96522. 

5,060 Views
L4 Transporter

i will do the upgrade on the change window.

can you please explain about below in more detail please

 

 PAN-92958 where the firewall isn't archiving and rotating /var/on file.

5,014 Views
Community Manager

https://docs.paloaltonetworks.com/pan-os/8-0/pan-os-release-notes/pan-os-8-0-addressed-issues/pan-os...

 

 

Fixed an issue where disk utilization increased unnecessarily because the firewall did not archive and rotate the /var/on file, which therefore grew to over 40MB.p> 

5,001 Views
L4 Transporter

MAny thanks Reaper 

4,973 Views
L1 Bithead

So, the solution is to upgrade to a new version?

We have had this issue in every version we have updated, and after the update the disk issue comes back months later. Every time we open a ticket, Palo alto support tells that the solution is to upgrade to a new OS version. In our case updating to a new version will fix the issue just temporarily.

 

4,954 Views
L7 Applicator

@KatiaNunez,

Please create a discussion on the 'General Topics' specfic to your issue. Include your model and software version along with the output of 'show system files'. 

4,952 Views
Ask Questions Get Answers Join the Live Community
Labels