topic Re: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices. in General Topics

Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

nikoolayy1 — Mon, 14 Jun 2021 13:52:40 GMT

When the Palo Alto partition is full depending on which partition is full differen issues may happen. The Palo Alto versions like 9.1.x and newer have much less issues with disk space like the 8.1 version, so an upgrade to newer version may help in many cases.

1. Generally I have seen issues becase of full disk where the HA communicaton is impacted, the commit operation fails, the GUI is not responsible and also the log correlation process is down as seen in the masterd log file in the managfment plane (usually this happens because of the log partition panlogs being full ). After cleaning the disk space use "show system software status" / "show system resources" (https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-cli-quick-start/cli-cheat-sheets/cli-cheat-sheet-device-management.html) to see if the process is still down and if needed restart it with "debug software restart process <name of the process>"

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClSjCAK

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClgZCAS

2. If the pancfg partition is full on Palo Alto firewalls or Panorama you will simply not be able to install new software versions of the panos or dynamic updates. In rare cases the GUI can be impacted.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLSJCA4&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000008V5bCAE&lang=en_US%E2%80%A9

3. An issue where the root partition is full you can only delete core files or system or process debug logs but if the issue persists then open a TAC case as they can access the Linux that Palo Alto uses as a core operational system and clean things from there.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRXCA0

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PMznCAG

4. In some rare cases a partition like the /dev/shm can be full and a reboot of the firewall could be needed. If the issue is still there the ultimate solution for non hardware palo alto issues is saving the config to external storage then factory default reset of the firewall and again importing the the config(the TAC does this many times).

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CldXCAS

https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/firewall-administration/reset-the-firewall-to-factory-default-settings.html

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRcCAK

Re: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

kiwi — Thu, 10 Jun 2021 08:53:02 GMT

Hi @nikoolayy1 ,

This is awesome ! Thanks for your contributions !

Cheers,

-Kiwi!

Re: Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

Jatin.Singh — Mon, 16 Aug 2021 23:57:15 GMT

Hi,

Our /dev/shm is at 97% after upgrading to version 10.0.6,what is the reason for this?

We have rebooted the device in past and it used to fix it.

However this time it is still high

Re: Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

nikoolayy1 — Tue, 17 Aug 2021 19:31:42 GMT

Because /dev/shm is used for temporary memory storage do you see any other partitions being full?

Maybe also check if your hard disk is ok?

How to Run a Hard Disk Drive Performance Test - Knowledge Base - Palo Alto Networks

I did not see any known bugs for /dev/shm for your version but maybe 10.x is too heavy for your device because of the all new features like ML machine learning etc. If you are using hardware appliance did you check if this is a supported version by device? If it is virtual edition do you see any system logs that may indicate that more hard this is needed by the virtual edition similar to PA-VM 4GB Root Partition 100 Percent Full. - Knowledge Base - Palo Alto Networks ?

Add Additional Disk Space to the VM-Series Firewall (paloaltonetworks.com)

Re: Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

Jatin.Singh — Wed, 18 Aug 2021 02:21:11 GMT

It is VM-300

I checked the device on auto assistant tool and it throws the following warming

> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.0G  4.3G  2.4G  65% /
none            7.7G   76K  7.7G   1% /dev
/dev/nvme0n1p5   16G  4.7G   11G  32% /opt/pancfg
/dev/nvme0n1p6  7.9G  1.9G  5.7G  25% /opt/panrepo
tmpfs           4.8G  4.7G  179M  97% /dev/shm
cgroup_root     7.7G     0  7.7G   0% /cgroup
/dev/nvme0n1p8   21G   13G  7.3G  64% /opt/panlogs
tmpfs            12M     0   12M   0% /opt/pancfg/mgmt/ssl/private

Rest of the disk space looks alright

Another thing I noticed the swap is zero used and free, is this normal?

What would be your recommendation on this? Can this be ignored as we are receiving no issues from it yet.

Re: Knowledge sharing: Troubleshooting and investigating full hard disk and full partitions (logs, config, root, etc.) issues on Palo Alto devices.

nikoolayy1 — Wed, 18 Aug 2021 08:13:30 GMT

I suggest checking with the support just in case (it could normal thing in the new versions but better they to confirm this) as you seem not the only one with such issues with the new 10.6:

LIVEcommunity - How to clean up /dev/shm - LIVEcommunity - 411602 (paloaltonetworks.com)

You also may check for memory alarms in the systems log or if there is a process with memory leakage. Also 10.0.7 is out so better test if upgrading to it if there is the same issue:

PAN-OS 10.0.7 Addressed Issues (paloaltonetworks.com)

The SWAP being 0 means that there is no disk space that can be used for memory extension if needed, so this is reverse to the /dev/shm and it could be because of /dev/shm being full there is less memory available and the SWAP gets to overused because of this. With most linux systems /dev/shm only uses real memory and if it gets full the SWAP is not used to extend /dev/shm but with Palo Alto and the relation between them only the TAC may tell this.

How to Interpret: show system resources - Knowledge Base - Palo Alto Networks

If the support tells you this is a bug or normal behavior, please share it with us.