topic Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use in General Topics

Disk usage for /opt/pancfg exceeds limit, 100 percent in use

PA_nts — Tue, 05 Mar 2024 09:36:41 GMT

Hi All,

running panorama 10.1.10-h2 VM in Azure with 4 managed VM100 FWs.

keep getting these alerts and not sure what else i can do. 'Disk usage for /opt/pancfg exceeds limit, 100 percent in use'

what i have done thusfar: - deleted any old panos versions / dynamic updates / gp clients etc etc - there is nothing more to delete that wont break the system. deleted snapshots also but these are quite small in comparison so doesn't make much of a difference.

padmin@panorama> show system disk-space
Filesystem Size Used Avail Use% Mounted on
....

/dev/sda5 24G 23G 238M 99% /opt/pancfg
...

if i run a 'request mongo show databases instance mgmt' it does not show the size of the mongodb. but there seems to be a few.

padmin@panorama> request mongo show databases instance mgmt

mon_summary_2024-02-29
pancfg
mon_summary_2024-02-26
mon_summary_2024-02-28
mon_summary_2024-02-20
mon_summary_2024-02-24
pan_rulehit
mon_summary_2024-02-21
admin
mon_summary_2024-02-27
mon_summary_2024-02-25
pan_devmon
pl_azure
mon_summary_2024-03-02
mon_summary_2024-02-22
mon_summary_2024-03-01
pan_warnings
panmgmtcache
mon_summary_2024-03-05
config
panreplay
pan_oplog
mon_summary_2024-03-03
local
mon_summary_2024-03-04
mon_summary_2024-02-23

padmin@panorama>

can some be deleted? and if so how?

i cannot do show system 'pancfg-directory-usage' because of not avail in 10.1

any ideas how to see the size of the mongodb's or how to clear more space?

thanks in adv.

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

JayGolf — Wed, 06 Mar 2024 05:53:54 GMT

Hi @PA_nts ,

Can you try the following command to clean up pancfg:

>debug pancfg-directory-usage set optimize-database-usage

If not, are you able to upgrade to the 10.2 preferred 10.2.8? 10.2.8 addresses disk space usage issues.

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

PA_nts — Wed, 06 Mar 2024 07:48:42 GMT

Hi JayGolf,

I am in a catch-22 situation..

this command is only avail in 10.2 - but i cannot download anything at the moment not even a content update.

>debug pancfg-directory-usage set optimize-database-usage

there is a fix in 10.1.12 also.. but same issue.. cannot download the image to perform the upgrade

at the moment i cleared out everything.. i only have 10.1.0 (base image) and 10.1.10-h2 images downloaded, 3 content updates (av, threat and wildfire) and my snapshots are less than 50mb total.

PAN-232800	Fixed an issue where critical disk usage for /opt/pancfg increased continuously and the system logs displayed the following message: Disk usage for /opt/pancfg exceeds limit, <value> percent in use .

thanks

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

PA_nts — Thu, 07 Mar 2024 13:22:38 GMT

Ok so update on this issue.. after exhausting all avenues with deleting images, content updates, config etc.. no luck.

in the end ran the following command to clear out the mongo DB which resolved the issue.. disk-space now down to 54%

panorama>debug mongo clear instance mdb
Are you willing to delete the data? This operation can not be reverted (y or n)

Delete data '' under path: /opt/pancfg/mgmt/mongo/data
Process 'mdb' executing STOP

panorama>

note when running this it will stop the mdb services.. so you need to verify and if stopped, need to actively start it up again.

panorama>show system software status

Process mdb stopped (pid: -1) - User Stop

panorama>

panorama> debug software restart process mdb

Process mdb was restarted by user NTT_Admin
panorama>

Will keep an eye on it for now.

thanks

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

A.Popielski — Mon, 19 Aug 2024 09:49:53 GMT

Hi,

I'm running into this issue now. Do you know what this command actually clears? Did you this in production during work hours? The command sounds pretty scary and I'm not too sure if it will brakre annything.

Thanks

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

PA_nts — Mon, 16 Sep 2024 09:03:00 GMT

I'd suggest to log a TAC for this if you can.. in my instance the tac engineer had to connect to the root OS to check the elastic seach health and found unassigned 'shards' on the es health.. which they had to clear manually. this process can only be performed by the tac as we as normal admins dont have permissions to log into the pan-os root.

Re: Disk usage for /opt/pancfg exceeds limit, 100 percent in use

A.Popielski — Mon, 16 Sep 2024 09:10:49 GMT

That's what I had to do. They had to get on it and use their TAC login, it took a few hours, but it clear the disk on panorama. We just rejoined the call once it was done.