PA-220 disk space issues

[skmurawski]

As most with a PA-220 have experienced, regardless version running (currently latest 10.2.3-h2) root partition fills up all the time and have to run the disk-usage cleanup commands manually and tried enabling aggressive-cleaning as well, it just continuously hits level/ But it just fills up all the effin time:-)

With the root disk issues seemingly persistent (since at least the high 9 versions) and with the PA-220 only containing a measly 32GB, apart from the disk issues we are perfectly happy with the PA-220 and have a handful across our network, so does anyone know if there is a storage upgrade option? just want to be able to have a bigger eMMC storage in it if possible, either by buying and installing or any Palo support routes that would enable a swap of storage?

Any ideas?

Thanks.

[jasonwald]

In my opinion you have two options.

1. Downgrade to 10.1.X

2. Abandon the platform and upgrade to PA-440 

[jasonwald]

There is no storage upgrade option. The hardware has been announced end of life and it is indeed on life support at this time. Start planning your hardware migration plan.

[Ed_Brown]

We are just going to upgrade to PA-440s for one situation but for the other PA-220s, we are looking at other alternatives such as moving the traffic to a different set of existing firewalls or possibly a different vendors product.

[JoshRountree]

I know you said it in your reply, but I just wanted to make clear, this problem isn't an issue in 10.1.X ?

We have several 220s and aren't able to upgrade to 440s just yet.

[Ed_Brown]

This problem DOES exist in 10.1.x. Then I heard the issue didn't exist in 10.2.6 so I moved to that and while less frequent, does still happen.

[jasonwald]

I have about 15 220's now running 10.1.11-h4 and have zero disk space issues. Just happens on my 10.2.X 220's of which I only have one left that i haven't either downgraded code on or replaced. Perhaps you are running something on your 10.1 220's that I am not. Not sure.

I hope to have all 220's out of my environment late next week!

[Daryl_Cruz]

We raised this issue to TAC, as per him, the issue was already fixed in 10.2.6 and later version however, if you encountered the same you must perform the aggressive cleaning to reduce the root disk space and it should be working. if you planning to upgrade to version 10.2.6 I suggest enable the aggressive cleaning  and set a threshold at least 94 after your upgrade.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS 

[Ed_Brown]

Have had aggressive cleaning going since 9.1 and confirmed was still in place on 10.2.6 but still getting disk issues although the message was in a different format.  Had the threshold set at 90%.  Ironically, the issue came up more on the passive in the HA pair than it did on the active.

[JMahannah]

I was still having this issue up to 10.2.6 and it appears to be resolved in 10.2.7-h3 (and 10.1.11-h4).  Went from 90% storage to around 68% for each 220.  

[JMahannah]

Update: 10.1.11-h4 has been pulled from a preferred release in favor of h5.  I didn't realize preferred releases could become non-preferred before a newer preferred release was released.  

[Hardik_Shah]

I don think issue was resolved for the update 10.2.7-h3
We have two firewalls VM-500 and they are both running version 10.2.7-h3 for long time now and still having issues with /dev/root  disk-space they both showing 

Firewall #1 

/dev/root 6.9G 5.9G 726M 90% /
none 14G 108K 14G 1% /dev
/dev/sda5 16G 6.1G 9.0G 41% /opt/pancfg
/dev/sda6 7.9G 1.8G 5.7G 24% /opt/panrepo
tmpfs 12G 9.6G 1.8G 85% /dev/shm
cgroup_root 14G 0 14G 0% /cgroup
/dev/sda8 21G 9.9G 9.8G 51% /opt/panlogs
tmpfs 12M 44K 12M 1% /opt/pancfg/mgmt/ssl/private

Firewall # 2

Filesystem Size Used Avail Use% Mounted on
/dev/root 6.9G 6.0G 534M 93% /
none 14G 116K 14G 1% /dev
/dev/sda5 16G 5.9G 9.1G 40% /opt/pancfg
/dev/sda6 7.9G 2.4G 5.1G 32% /opt/panrepo
tmpfs 12G 9.6G 1.8G 85% /dev/shm
cgroup_root 14G 0 14G 0% /cgroup
/dev/sda8 21G 9.9G 9.8G 51% /opt/panlogs
tmpfs 12M 44K 12M 1% /opt/pancfg/mgmt/ssl/private

[GOMEZZZ]

Hi Daryl,

we are facing the same issue on the pa-220 disk space in filling op all the time leading to ha failures.

First tac pointed to bug 

PAN-231169 that supposed to be fixed 10.2.7 h3 what was the version we where running. The. They said we need to upgrade to 10.2.8.  We had the same issues on the 10.1 release. Officially the Pa-220 is supported for the 10.1 and 10.2 release nd the hardware support is still valid for a couple of years. Palo should fix this. We have been mailing back and forward with palo for about 2 months now without any  solution.  Did you find a solution for it? We now running a scheduled script to run the cleanup command every day what is crazy if you ask me. 

[JMahannah]

I ran into a few solutions for these errors that were causing intermittent outages.  I have a fleet of PA-220s

1) Beware of your upgrade path.  There are some landmines on the way to 10.2.8.  I went 10.1.2 - 10.1.6 - 10.1.7 - 10.1.11 -  10.1.11-h3.  10.1.11-h3 is working great for most of my appliances.  For the others that I inherited at 10.2.1, I continued to 10.2.0 - 10.2.1 - 10.2.2 - 10.2.3 - 10.2.6 - 10.2.7 - 10.2.7-h3. There are some you can't jump from/to.  Make sure you read all the release notes (and especially search for notes referencing the 220).  

2) On some devices, I got stuck upgrading and either had to hard reset or wipe and reload.  The space issues only resolved when wiping and allowed me to continue on the upgrade path (back up and export your named configs!).   

3) Intermittently, if storage gets overrun on the path up, run these commands (note "\r" is for enter or new line in SecureCRT):

• These help with ROOT folder:
  • debug software disk-usage aggressive-cleaning enable
  • debug software disk-usage cleanup deep threshold 90 
  • delete debug-log mp-log file *.old \r delete debug-log mp-log file *.1 \r delete debug-log mp-log file *.2 \r delete debug-log mp-log file *.3 \r delete debug-log mp-log file *.4 \r
• These help with the PANCFG folder
  • delete content cache old-content \r
  • show max-num-images \r set max-num-images count
  • debug software restart process management-server \r (note that this will reset all current sessions)

Hang in there! Sometimes it was a daily struggle to keep the Firewalls running while waiting for the next upgrade, patch, and outage window.  You might need to run these commands multiple times each day to keep the storage from overrunning.  

[GOMEZZZ]

Hi,

thank you for the reply. We had indeed a few that had some issues on the upgrade path but got there eventually. But we are still facing the space issue. After some time the disk just fills up and causes ha to crash. 
how do you stop this? As said we are now daily running the cleanup commands. 
how did you stop it from filling up?

[JMahannah]

Personally, I would go to the most recent 10.2 (10.2.8 the day of this post).  You might have to perform factory resets all along the path or at least whenever it fails.  They each take about 30 minutes to install and if they hang for much more than that, hard reset.  Once you get to your target, you might need another factory reset and reload your config to stabilize after all the junk upgrades. 

Then start budgeting for PA440s!