topic Re: High Disk Space Usage on /opt/pancfg partition in Next-Generation Firewall Discussions

High Disk Space Usage on /opt/pancfg partition

Ankit1Singh — Thu, 14 Sep 2023 12:50:34 GMT

Hi Mates,

I am getting alerts on /opt/pancfg utilizing 90%.

How ever I deleted the old, downloaded software and dynamic updates (around 1.5Gb file) but still space utilization is same as 90% using below KB.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLSJCA4

Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.8G 3.4G 224M 94% /
/dev/md5 7.6G 6.4G 839M 89% /opt/pancfg
/dev/md6 3.8G 2.9G 701M 81% /opt/panrepo

Why it showing same space after deleting the files and how to decrease the space on this?

Re: High Disk Space Usage on /opt/pancfg partition

BPry — Thu, 14 Sep 2023 13:00:48 GMT

@Ankit1Singh,

Something to keep in mind with 'show system disk-space' is that these numbers aren't polled when the command is issued; the firewall looks at utilization on a regular basis and simply displays the last numbers that were gathered when this command is issued.

How long did you wait before checking disk-space again? If it was immediately after removing the files, I wouldn't expect it to show updated values right away.

Re: High Disk Space Usage on /opt/pancfg partition

Ankit1Singh — Thu, 14 Sep 2023 13:15:20 GMT

Thanks for the reply. I have waited for 5 minutes but still it shows same %. How long should I wait to see the change in values?

Before deleting the software files. available space was 839M but after I deleted old sw-images available space shows 803M.

Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.8G 3.4G 224M 94% /
/dev/md5 7.6G 6.4G 839M 89% /opt/pancfg
/dev/md6 3.8G 2.9G 701M 81% /opt/panrepo

After deleting the software files.

Filesystem Size Used Avail Use% Mounted on
/dev/md2 3.8G 3.4G 224M 94% /
/dev/md5 7.6G 6.4G 803M 89% /opt/pancfg

Re: High Disk Space Usage on /opt/pancfg partition

pbezouska — Fri, 22 Sep 2023 20:14:11 GMT

I have the same problem. In my situation, it's caused by pancfg mongoDB which resides in the same partition as SW updates and config revisions (on Panorama in Panorama mode). You can check if that is the case with you by issuing this command:
request mongo show databases instance mgmt
This should give you list of mongo DBs with it's respective sizes.
I guess, normally, these aren't large but in my case, it's taking most of the partition's allocated space.
TAC is useless, they can't even navigate through Linux FS. So I had to investigate on this myself since it was crashing the management server.
I figured out that the abnormally large collection in pancfg DB (in my case) is a "tags_ip_rev" collection that seems to store IP-Tag revisions for some odd reason...
At least there's a command that drops the whole collection and it seems to have no negative impact
debug mongo drop collection tags_ip_rev database pancfg instance mdb
I have to use this command every other day so the appliance won't go into zombie mode without doing failover (pretty common thing on panorama)... This instantly clears all the space taken by the pancfg DB but it doesn't really solve anything.
So I'm waiting for TAC until they find the solution which could take years considering that they probably didn't figure out that there's a mongoDB running on the backend.
Even though I've given all these clues to them some time ago.

Re: High Disk Space Usage on /opt/pancfg partition

Ankit1Singh — Fri, 22 Sep 2023 20:21:48 GMT

Thanks @pbezouska for the reply, in my case its a Firewall in HA.

Where you have checked and confirmed it's caused by pancfg mongoDB. Also there is no crash in my platform.

Model – PA3050

PANOS – 8.1.13

Also, can you help on how can I navigate to /opt/pancfg to see what are the files inside this partition?

Linux commands to navigate to /opt/pancfg.

Re: High Disk Space Usage on /opt/pancfg partition

pbezouska — Sat, 23 Sep 2023 01:05:55 GMT

Hah, that's pretty old software if you'd ask me.
My problem is specific for all latest versions above 10.2.0, even the latest 11.0.2-h2 is impacted. And also Panorama PANOS, not FW.
It could be a number of different reasons then, I had this happen on me before and it involved a session with TAC to clear something manually since you can't get root access on PANOS yourself.
Specific commands that I used when trying to find out what's taking up space in that partition are available only in 10.2 and above.
What I'd do in your case would be try decreasing the disk usage by booting into maintenance menu, perform FSCK and probably wiping panlogs, boot PANOS and if you have enough space, squeeze in an upgrade to recommended version.

Re: High Disk Space Usage on /opt/pancfg partition

Pras — Mon, 08 Jan 2024 08:06:13 GMT

@pbezouska Hello, How do you perform FSCK and wiping panlogs ?

Thanks,

Pras