How and When to Clear Disk Space on the Palo Alto Networks Device

Printer Friendly Page

Details

The Palo Alto Networks device deletes the oldest log data when the logdb-quota is reached. The device purges logs based upon categories seen in show system logdb-quota.

Refer to When are Logs Purged on the Palo Alto Networks Devices? for behavior of purging on different platforms.

 

The root partition can become full, requiring manual file deletion. If the root is full, the device cannot to perform maintenance tasks such as content installs (AV, APP/Threat, URL, DB) or generate tech support files.  To check the status of the root partition, use the show system disk-space command. Core files consume large amounts of disk space: show system files. Delete large core files: delete core management-plane file <filename>.

 

Use these commands to view and delete core files:

 

> show system disk-space
 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3            3.8G  3.8G    0 100% /
/dev/sda5            7.6G  3.4G  3.8G  48% /opt/pancfg
/dev/sda6            3.8G  2.7G  940M  75% /opt/panrepo
tmpfs                493M  36M  457M  8% /dev/shm
/dev/sda8              51G  6.6G  42G  14% /opt/panlogs

 

  1. Check the output of show system file to see core files using up a large amount of disk space.
    > show system files
    /opt/dpfs/var/cores/:
    total 4.0K
    drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo
    
    /opt/dpfs/var/cores/crashinfo: total 0   /var/cores/: total 115M drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo -rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz -rw-rw-rw- 1 root root  51M Jun 12 13:39 core.20053   /var/cores/crashinfo: total 16K -rw-rw-rw- 1 root root 15K Jun 10 20:15 devsrvr_4.0.3-c37_0.info

     

  2. Delete unnecessary core files:
    > delete core management-plane file devsrvr_4.0.3-c37_1.gz
    (this example deletes a device server core file from the management-plane).
    Report deletion can be done from the command line as well.  To delete a set of summary reports starting with 864:
    > delete report summary scope shared report-name predefined file-name 864*
  3. Delete rotated files and files with extention .old as follows. These files contain monitoring details and service related logs on the firewall. Hence they can be deleted safely if you don't need them. If TAC investigates an ongoing issue,  you may prefer to keep them until you upload the tech support file to the case manager. 

     

    > delete debug-log mp-log file *.1
    > delete debug-log mp-log file *.2
    > delete debug-log mp-log file *.3
    > delete debug-log mp-log file *.old
    

 

       

owner: bpappas

Comments

If none of these options work, what then?

Open a case with the palo alto support. They can access root and delete more.

anohter log for disk space

sh: cannot create temp file for here document: No space left on device

deleting old system logs helped

How often an PA admin need to do this kind of clean up ?  In my case system  disk space is getting full frequenlty & I have to manually  delete files like the solution given in this article. Earlier it was told that it is bug issue in PAN OS 7.1.3 so we upgraded to 7.1.5 but it looks issue is not resolved. 

Hi @Srivastava

it depends: are you encountering issues (error messages) regularly? if there's no errors or specific issues (not being able to create a core file due to lack of available space for example) there should also not be the need to delete anything.

are you applying the above deletes when your disk is at 100%, or less ? (if less, this may not be necessary)

We see this issue frequently as well and are rarely about the 14% free mark.  we run 7.1.7 and haven't seen the issue resolved. 

But is there a way to set this cleaning automatical? Or can we anywhere define a maxi value for some log files? Because so i must clean every two weeks the file system of the pa200.

hi @clonesheep

as mentioned in the first line of the article, log files are purged automatically (first in first out, based on quota)

 

This article is in regards to the root ( / ) filling up, this occurs typically over a long period of time and when some processes have crashed in the past (core files)

The mp-log files rotate out automatically (hence the .1 , .2 , ... extention) , but it could be helpful to temporarily clean up older log files in case you need some additional free space to perform an upgrade or for other reasons

 

If you're encountering more accute issues, please reach out to support

hi @reaper

 

can i change anywere the time for this "mp-log files rotate out automatically"? Because by us it is not enoght space and we get more then 95% and then it gives a warning. so any kind of the mp-log log file will be ~10mb and then it take a new file.. .1 .2 .3 .4 .old but i cant see a automism what will delete these files?!

Hi @clonesheep

 

There's 2 distinct types of logfiles: the logdb, which is one giant database containing all the 'firewall' logging like traffic and threat and then there are the device logs that reside on the hard drive's filesystem

 

The logdb has a quota and also has alerting which you can enable to signal you when a log is exceeding a certain level. the logdb will automatically purge old logs once a log reaches about 98% of capacity

If you're seeing alerts that the 'system' log has passed 95%, this is the logdb system, not the mp or dp .log files

If you want the 'system' logs to take up less space, you can simply change the quota via Device > System > Management > Logging and Reporting settings

 

The device logs rotate to .n once they reach their max size (as determined by our development engineers), after 4 backups have been created the oldest one (.4) will start to get purged once a new .1 needs to be created. These logs live in a controlled ecosystem so under normal operating conditions you won't need to clear them out

 

The logdb 'system' log contains messages in regards to the firewall system while the mp and dp logs contain info at the device operating system level

 

hope this helps

@reaperthanks that helps.

 

 Try also:
debug dataplane packet-diag clear log log

to clear disk space after troubleshooting.

hi @reaper Thanks for all your contributions!
Can I ask you a question?

In my case the partition /dev/sda3 is 83%

 

 

After executing the show system files command I get the following:

 

 

> show system files
/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Nov 11 2016 crashinfo

/opt/dpfs/var/cores/crashinfo:
total 0

/var/cores/:
total 349M
drwxrwxrwx 2 root root 4.0K Nov 11 2016 crashinfo
-rw-rw-rw- 1 root root 876M Oct 23 11:17 core.3298

/var/cores/crashinfo:
total 0

Can I delete the core.3298 file that weighs 876M?

I'm not sure if deleting it could cause some flaw or error in my firewall

 

Thanks! 

 

"The man with a new idea is a Crank until the idea succeeds." (Mark Twain)

 

hi @SalvadorCordova

 

a core file is a flat (text) file that contains the memory dump of a process that crashed

 

it usually starts out as a core.number and once completed is changed to the process name of the process that crashed, so this core file is incomplete

it is safe to delete the file,but export it off the system first (> tftp/scp export core-file management-plane from * to user@host/path) 

, it is there because something bad happened, so keep an eye on your sytem to ensure no new core files occur

Thanks @reaper

 

Bless you!

I have gone back and forth on disk space with tac and with the live community. I had tac clean up the log files before and that didn't gain me much. I am trying to prepare for an upgrade to OS 8.x this summer and I want to make sure I do not run out of diskspace and also not break anything trying to do some clean up

I guess I want to know if these need to be cleaned up and verify what they are

 

/dev/md6 3.8G 3.2G 454M 88% /opt/panrepo - PAN-OS Image repository (are these files that are located device]software)? Do the old OS's need to be deleted? I asked tac once before and they said no and then I asked at another date and they said yes - so I am confused.

 

/dev/md5 7.6G 3.3G 4.0G 46% /opt/pancfg - PAn-OS configuration is this the device config files and do they affect

 

 

Hi @jdprovine! the panrepo contains  OS images you previously downloaded and are marked as such in the GUI in the device tab under Software

 

deleting 'stuff' here is both a yes and no: you need to be aware of what is currently installed and what was previously installed (one version back): you can delete everything that is not required to be present for a rollback

 

eg: if you previously had your system om 8.0.1 and are now on 8.0.5, you can delete all 7.1, 7.0, 6.1,... images,

you can also delete 8.0.2, .3, .4, but you cannot delete 8.0.0 as it serves as a base image for your currently installed version, nor 8.0.1 as it is used in the rollback partition

 

if you previously had 7.1.10 and now 8.0.5, you can delete everything except 7.1.0, 7.1.10, 8.0.0 and 8.0.5 (because these 4 are in use for the currently running system and the rollback partition

 

to clean up space in the pancfg directory, you can go ahead and remove old config files

> delete config saved

but unless you made several backups a day for a long time, there should not be a lot there (also usulally no need to clear out that directory unliss it's at 95%)

 

@reaper

I have cleaned it up and am all ready to go. Thanks for the update reaper

In our case there were packet diag logs
that is not viewable in GUI and only visible via CLI, though I think this is because we recently updated to PAN-OS 8 from PAN-OS 7.1.7. The command below will be helpful.

 

FW(activ)> debug dataplane packet-diag show setting ----> to verify if there is one

FW(active)> debug dataplane packet-diag clear log log -----> to clear the logs

Hi all.

I'm still having same issue on PA-200 devices running 8.0.6 and 8.0.7 with no luck.

In my case, the root partition is running 95% of disk usage and system files are less than 4K.

 

Any final solution on getting Pa firewalls less than ~85% of disk usage rather than call PA support for clean up?

@edgutier, were these PS-200 units upgraded from PAN-OS 7.1.x  or older? If so, then the log files that were piling up in the past may still be there and the only way is to get support to log into the unit with root access and clear out those directories.

 

Hi @jdelio, thank you for your response.

I have upgraded the PA-200 Firewalls with PA support from 7.0 to 8.0.6 and 8.0.7 and the PA engineer performed clean up during upgrade process, but now the two firewalls have 95% of disk usage of the root partition and there are no any system files greater than 4K.

 

@edgutier, the root getting full is normally a log file that needs to be wiped, but can only be accessed via Root/Support access.

I would recommend contacting support again to have root access/clean the access log issue.

Hi @jdelio,  thank you for your response.

I have called PA support to resolve this issue in the past but this problem is happening over and over again and I cannot call PA support every 2 or 4 weeks.....

I need a final solution for disk space issue on PA-200 devices.

 

@edgutier, I am sorry to hear that this is such a repetitive issue for you. Are these the same devices that you are having issues with? The known issue with the disk space on the PA-200 units should be resolved with PAN-OS 8.0. But this is future issues, if they were pre-existing, then that is why I recommended contacting support. 

 

If these are the same devices over and over again, then there is something else that needs to be done to prevent this, as having to call up every month (at best) for the same devices is too much.

 

I am sure that if you have a case with Support (TAC), that if you let them know, and escalate this issue, they should be able to find a permamnent solution for you.

Hi @jdelio, yes, these are the same devices I have the repetitive issue. Both are PA-200 devices, one running 8.0.6 and other 8.0.7.

I had to open a ticket with TAC to see if there's a permanent solution.

We have the same issue with our PA-200. OS 8.0.7 and 8.0.8.

After cleanup (see show system disk-space below) the filesystem fills up to 95% after 1-2 weeks.

 

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.9G  1.6G  232M  88% /
/dev/sda5             6.6G  2.2G  4.2G  34% /opt/pancfg
/dev/sda6             1.9G  859M  974M  47% /opt/panrepo
tmpfs                 1.2G  116M  1.1G  10% /dev/shm
/dev/sda8             2.4G  1.7G  640M  73% /opt/panlogs
tmpfs                  12M     0   12M   0% /opt/pancfg/mgmt/lcaas/ssl/private

 

Could you please confirm if there's any solution on this issue rather than call PA support? (like edgutier  wrote before)

@Clermont, This is not anything that I am aware of, nor do I know of a resolution.

This issue needs to be reported to support (TAC) so this can be escalated and a resolution found. Because if this is happening to 2 people then it is more than likely happening to others and the best way to handle it would be to have support document this. If this continues, then this needs to be escalated and a permanent solution found.

I ran the command, show system disk-space

 

Filesystem Size Used Avail Use% Mounted on
/dev/md3 3.8G 3.5G 106M 98% /
/dev/md5 7.6G 3.4G 3.8G 47% /opt/pancfg
/dev/md6 3.8G 1.5G 2.2G 40% /opt/panrepo
tmpfs 2.0G 116M 1.9G 6% /dev/shm
cgroup_root 2.0G 0 2.0G 0% /cgroup
/dev/md8 198G 134G 55G 72% /opt/panlogs
tmpfs 12M 0 12M 0% /opt/pancfg/mgmt/lcaas/ssl/private

 

Here's the result. The largest file is 7.5M. I wonder where that 3.5G is?

 

/opt/var.dp2/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Feb 3 10:42 crashinfo

/opt/var.dp2/cores/crashinfo:
total 0

/opt/var.dp1/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Feb 3 10:42 crashinfo

/opt/var.dp1/cores/crashinfo:
total 0

/opt/var.dp0/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Feb 3 10:42 crashinfo

/opt/var.dp0/cores/crashinfo:
total 0

/opt/var.cp/cores/:
total 4.0K
drwxr-xr-x 2 root root 4.0K Feb 3 10:44 crashinfo

/opt/var.cp/cores/crashinfo:
total 0

/var/cores/:
total 7.5M
drwxr-xr-x 2 root root 4.0K Feb 23 09:06 crashinfo
-rw-r--r-- 1 root root 7.5M Feb 23 09:15 mgmtsrvr_8.0.6-h3_0.tar.gz

/var/cores/crashinfo:
total 8.0K
-rw-rw-rw- 1 root root 6.8K Feb 23 09:06 mgmtsrvr_8.0.6-h3_0.info

/opt/panlogs/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Feb 24 11:55 crashinfo

/opt/panlogs/cores/crashinfo:
total 0