Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

L6 Presenter

When the Palo Alto partition is full depending on which partition is full differen issues may happen. The Palo Alto versions like 9.1.x and newer have much less issues with disk space like the 8.1 version, so an upgrade to newer version may help in many cases.

 

 

 

 

1. Generally I have seen issues becase of full disk where the HA communicaton is impacted, the commit operation fails, the GUI is not responsible and also the log correlation process is down as seen in the masterd log file in the managfment plane (usually this happens because of the log partition panlogs being full ). After cleaning the disk space use "show system software status" / "show system resources" (https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-cli-quick-start/cli-cheat-sheets/cli-cheat-sheet...) to see if the process is still down and if needed restart it with "debug software restart process <name of the process>"

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClSjCAK

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClgZCAS

 

 

 

 

 

2. If the pancfg partition is full on Palo Alto firewalls or Panorama you will simply not be able to install new software versions of the panos or dynamic updates. In rare cases the GUI can be impacted.

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLSJCA4&refURL=http%3A%2F%...

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000008V5bCAE&lang=en_US%E2%80%A...

 

 

 

 

 

3. An issue where the root partition is full you can only delete core files or system or process debug logs but if the issue persists then open a TAC case as they can access the Linux that Palo Alto uses as a core operational system and clean things from there.

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRXCA0

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PMznCAG

 

 

 

 

 

 

4. In some rare cases a partition like the /dev/shm can be full and a reboot of the firewall could be needed. If the issue is still there the ultimate solution for non hardware palo alto issues is saving the config to external storage then factory default reset of the firewall and again importing the the config(the TAC does this many times).

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CldXCAS

 

https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/firewall-administration/reset-the-firewall...

 

 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRcCAK

 

5 REPLIES 5

Community Team Member

Hi @nikoolayy1 ,

 

This is awesome ! Thanks for your contributions !

 

Cheers,

-Kiwi!

 
LIVEcommunity team member, CISSP
Cheers,
Kiwi
Please help out other users and “Accept as Solution” if a post helps solve your problem !

Read more about how and why to accept solutions.

L3 Networker

Hi, 

Our /dev/shm  is at 97% after upgrading to version 10.0.6,what is the reason for this?

We have rebooted the device in past and it used to fix it.

However this time it is still high

 

Because /dev/shm is used for temporary memory storage do you see any other partitions being full?

 

 

Maybe also check if your hard disk is ok?

 

 

How to Run a Hard Disk Drive Performance Test - Knowledge Base - Palo Alto Networks

 

 

 

I did not see any known bugs for /dev/shm for your version but maybe 10.x is too heavy for your device because of the all new features like ML machine learning etc. If you are using hardware appliance did you check if this is a supported version by device? If it is virtual edition do you see any system logs that may indicate that more hard this is needed by the virtual edition similar to PA-VM 4GB Root Partition 100 Percent Full. - Knowledge Base - Palo Alto Networks ?

 

 

Add Additional Disk Space to the VM-Series Firewall (paloaltonetworks.com)

It is VM-300

I checked the device on auto assistant tool and it throws the following warming

 

JatinSingh_0-1629253040232.png

 

> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.0G  4.3G  2.4G  65% /
none            7.7G   76K  7.7G   1% /dev
/dev/nvme0n1p5   16G  4.7G   11G  32% /opt/pancfg
/dev/nvme0n1p6  7.9G  1.9G  5.7G  25% /opt/panrepo
tmpfs           4.8G  4.7G  179M  97% /dev/shm
cgroup_root     7.7G     0  7.7G   0% /cgroup
/dev/nvme0n1p8   21G   13G  7.3G  64% /opt/panlogs
tmpfs            12M     0   12M   0% /opt/pancfg/mgmt/ssl/private

Rest of the disk space looks alright

 

Another thing I noticed the swap is zero used and free, is this normal?

 

JatinSingh_1-1629253149348.png

 

 What would be your recommendation on this? Can this be ignored as we are receiving no issues from it yet.

L6 Presenter

I suggest checking with the support just in case (it could normal thing in the new versions but better they to confirm this) as you seem not the only one with such issues with the new 10.6:

 

LIVEcommunity - How to clean up /dev/shm - LIVEcommunity - 411602 (paloaltonetworks.com)

 

 

 

You also may check for memory alarms in the systems log or if there is a process with memory leakage. Also 10.0.7 is out so better test if upgrading to it if there is the same issue:

 

PAN-OS 10.0.7 Addressed Issues (paloaltonetworks.com)

 

 

The SWAP being 0 means that there is no disk space that can be used for  memory extension if needed, so this is reverse to the /dev/shm and it could be because of  /dev/shm being full there is less memory available and the SWAP gets to overused because of this. With most linux systems   /dev/shm only uses real memory and if it gets full the SWAP is not used to extend /dev/shm but with Palo Alto and the relation between them only the TAC may tell this.

 

How to Interpret: show system resources - Knowledge Base - Palo Alto Networks

 

 

 

 

If the support tells you this is a bug or normal behavior, please share it with us.

  • 9811 Views
  • 5 replies
  • 3 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!