PAN-193452 (PaloAlto 220 exceeds disk usage capacity) - Is this fixed in 10.2.3 ?

[Michael.W]

We have PA220 firewalls and regularly get the disk usage capacity error. With the help of the support team and KB instructions we can clear down space, but the message starts appearing a few days later.

This is listed as a known issue (PAN-193452). Does anyone know if this is fixed in the latest release (10.2.3) and whether we are good to go ahead and push this out the PA220s? 

Reading through the 10.2.3 release notes, this issue is listed as both a Known Issue and in the Addressed Issue, so it is not clear whether this is fixed or not:

https://docs.paloaltonetworks.com/content/techdocs/en_US/pan-os/10-2/pan-os-release-notes/pan-os-10-...

https://docs.paloaltonetworks.com/content/techdocs/en_US/pan-os/10-2/pan-os-release-notes/pan-os-10-...

We have not yet upgraded to 10.2.3 as the previous version (10.2.2-h2) is still showing as the preferred version and in previous discussions with the support team they have stressed the importance of sticking with the preferred release listed on the support page:

https://live.paloaltonetworks.com/t5/Customer-Resources/Support-PAN-OS-Software-Release-Guidance/ta-...

It would be good to get confirmation on whether this is fixed in 10.2.3 and whether we should roll this out.

[pan219]

Sadly, it is not fixed in 10.2.3 as I am running into this issue every few days also (PA220, 10.2.3)

Besides this unfixed issue they introduced another showstopper-bug which makes it impossible to connect to GP with local user accounts on the palo using iOS - downgrading to 10.2.2.-h2 does not fix this mess.

[JeffAdams_T3]

I can tell you I am running 10.2.3 on my PA-220 and am still having the issue so it does not appear to be fixed.  Support recommended downgrading to 10.1.6-H6.  I ran command  "debug software disk-usage cleanup deep threshold 90", which gave me an error but did drop the system disk space utilization from 95% down to 91%.  It will, however, keep creeping back up.  That is my experience.

[jasonwald]

Still having this issue on 10.2.4 with PAN-220

show system disk-space

Filesystem Size Used Avail Use% Mounted on
/dev/root 3.8G 3.5G 113M 97% /
none 2.5G 60K 2.5G 1% /dev
/dev/mmcblk0p5 13G 2.9G 8.9G 25% /opt/pancfg
/dev/mmcblk0p6 3.8G 1.1G 2.6G 28% /opt/panrepo
tmpfs 2.5G 355M 2.2G 15% /dev/shm
cgroup_root 2.5G 0 2.5G 0% /cgroup
/dev/mmcblk0p8 3.7G 2.6G 957M 74% /opt/panlogs
/dev/loop0 111M 5.6M 100M 6% /opt/panlogs/wildfire/tmpfile
tmpfs 12M 36K 12M 1% /opt/pancfg/mgmt/ssl/private

debug techsupport duts run

/var depth 1 ====================================================

605M /var

/var depth 2 ====================================================

323M /var/appweb
254M /var/log
23M /var/lib
3M /var/cli
1M or less: 27 files

/var depth 3 ====================================================

310M /var/appweb/htdocs
184M /var/log/pan
28M /var/log/sa
22M /var/lib/rpm
22M /var/log/nginx
13M /var/appweb/sslvpndocs
11M /var/log/syslog-system.log-20230521
6M /var/log/syslog-system.log
3M /var/cli/help
1M or less: 97 files

/var depth 4 ====================================================

85M /var/appweb/htdocs/js
69M /var/appweb/htdocs/PAN_help
56M /var/appweb/htdocs/php
26M /var/appweb/htdocs/restapi-doc
26M /var/appweb/htdocs/styles
19M /var/lib/rpm/Packages
16M /var/appweb/htdocs/vendor
14M /var/appweb/htdocs/scripts
10M /var/appweb/sslvpndocs/global-protect
8M /var/log/pan/md_apps.log
8M /var/log/pan/dnsproxyd.log
8M /var/log/nginx/api_metrics.log
7M /var/log/pan/req_stats.log
7M /var/log/pan/routed.log
6M /var/log/pan/distributord.log
6M /var/log/pan/rgxgrpsl01dp00.dat
6M /var/log/pan/cryptod.log
6M /var/log/pan/l3svc_ngx_error.log
6M /var/log/pan/iotd.log
6M /var/log/nginx/error.log
6M /var/log/pan/sslvpn_ngx_error.log
6M /var/log/pan/logging-services.log
6M /var/appweb/htdocs/images
5M /var/log/nginx/access.log
5M /var/log/pan/dagger.log
5M /var/log/pan/useridd.log
5M /var/log/pan/devsrv.log
4M /var/appweb/htdocs/locale
4M /var/log/pan/pan_dhcpd.log
4M /var/log/pan/mp-monitor.log
4M /var/log/pan/configd.log
4M /var/log/pan/redis_useridd.log
4M /var/log/pan/redis_mgmt.log
4M /var/log/pan/syslog-ng.log
3M /var/log/pan/mprelay.log
3M /var/log/pan/contentd.log
3M /var/log/pan/device_telemetry.log
3M /var/log/pan/ha_agent.log
3M /var/log/pan/qtrace_routed.log
3M /var/appweb/htdocs/php-packages
2M /var/log/pan/md_out.log
2M /var/appweb/sslvpndocs/saml
2M /var/log/pan/paninstaller_content.log
2M /var/log/pan/threatvault.log
2M /var/log/pan/ikemgr.log
2M /var/log/pan/get_content_issue_msg.log
2M /var/log/pan/pan_cmd.log
2M /var/log/pan/sysd.log
2M /var/lib/rpm/Basenames
2M /var/log/pan/report_gen.log
2M /var/log/pan/device_telemetry_send.log
1M /var/log/pan/device_telemetry_minute.log
1M /var/log/pan/upgradelog_err_content
1M /var/log/nginx/sslvpn_access.log
1M /var/log/pan/md_detail.log
1M /var/log/pan/sysdagent.log
1M /var/log/pan/nf_conntrack.log
1M /var/log/pan/device_telemetry_cli.log
1M /var/log/pan/device_telemetry_day.log
1M /var/log/pan/device_telemetry_curl.log
1M /var/log/nginx/sslvpn_access.log.1.zip
1M or less: 456 files

/var depth 5 ====================================================

41M /var/appweb/htdocs/js/lib
32M /var/appweb/htdocs/js/pan
27M /var/appweb/htdocs/php/3rdParty
24M /var/appweb/htdocs/styles/login
15M /var/appweb/htdocs/php/misc
15M /var/appweb/htdocs/PAN_help/en
14M /var/appweb/htdocs/scripts/cache
11M /var/appweb/htdocs/PAN_help/fr
11M /var/appweb/htdocs/PAN_help/es
11M /var/appweb/htdocs/PAN_help/ja
10M /var/appweb/htdocs/js/3rdParty
10M /var/appweb/sslvpndocs/global-protect/portal
10M /var/appweb/htdocs/PAN_help/zh_TW
9M /var/appweb/htdocs/vendor/setasign
8M /var/appweb/htdocs/PAN_help/zh_CN
6M /var/appweb/htdocs/vendor/mongodb
5M /var/appweb/htdocs/php/locale
2M /var/appweb/htdocs/restapi-doc/redoc.standalone.js.map
2M /var/appweb/htdocs/php/device
1M /var/appweb/htdocs/restapi-doc/PA-220,multi-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220R,multi-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220,single-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220-ZTP,multi-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220R-ZTP,single-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220-ZTP,single-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220R-ZTP,multi-vsys-mode.yaml
1M /var/appweb/htdocs/restapi-doc/PA-220R,single-vsys-mode.yaml
1M /var/appweb/htdocs/php/include
1M /var/appweb/sslvpndocs/saml/images
1M /var/appweb/htdocs/images/treemenu
1M /var/appweb/sslvpndocs/saml/fonts
1M /var/appweb/htdocs/locale/ja
1M /var/appweb/htdocs/restapi-doc/PA-220-ZTP,single-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220R,single-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220-ZTP,multi-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220R,multi-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220,multi-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220,single-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220R-ZTP,single-vsys-mode.json
1M /var/appweb/htdocs/restapi-doc/PA-220R-ZTP,multi-vsys-mode.json
1M /var/appweb/htdocs/locale/fr
1M or less: 656 files

/var depth 6 ====================================================

27M /var/appweb/htdocs/php/3rdParty/tcpdf
24M /var/appweb/htdocs/styles/login/fonts
15M /var/appweb/htdocs/php/misc/namedschema
9M /var/appweb/htdocs/vendor/setasign/tfpdf
8M /var/appweb/htdocs/PAN_help/en/wwhdata
8M /var/appweb/sslvpndocs/global-protect/portal/fonts
7M /var/appweb/htdocs/js/pan/device
6M /var/appweb/htdocs/vendor/mongodb/mongodb
6M /var/appweb/htdocs/js/3rdParty/ext
6M /var/appweb/htdocs/js/pan/locale
6M /var/appweb/htdocs/js/lib/extjs3
4M /var/appweb/htdocs/PAN_help/fr/wwhdata
4M /var/appweb/htdocs/js/lib/pan-extjs3.js.map
4M /var/appweb/htdocs/PAN_help/es/wwhdata
3M /var/appweb/htdocs/PAN_help/zh_TW/images
3M /var/appweb/htdocs/js/3rdParty/CodeMirror
3M /var/appweb/htdocs/js/lib/panos-panos-react.js.map
3M /var/appweb/htdocs/PAN_help/ja/wwhdata
3M /var/appweb/htdocs/js/pan/network
2M /var/appweb/htdocs/js/lib/panos-panos-react.js
2M /var/appweb/htdocs/js/pan/base
2M /var/appweb/htdocs/js/lib/pan-extjs3.js
2M /var/appweb/htdocs/PAN_help/fr/images
2M /var/appweb/htdocs/PAN_help/zh_CN/images
2M /var/appweb/htdocs/PAN_help/es/images
2M /var/appweb/htdocs/scripts/cache/device.javascript
2M /var/appweb/htdocs/PAN_help/ja/images
2M /var/appweb/htdocs/PAN_help/ja/css
2M /var/appweb/htdocs/js/pan/objects
2M /var/appweb/htdocs/scripts/cache/mainui.javascript
2M /var/appweb/htdocs/js/lib/pan-extjs3-main-ext.js.map
1M /var/appweb/htdocs/PAN_help/zh_TW/css
1M /var/appweb/htdocs/PAN_help/zh_CN/css
1M /var/appweb/htdocs/PAN_help/es/css
1M /var/appweb/htdocs/PAN_help/fr/css
1M /var/appweb/htdocs/js/pan/acc2
1M /var/appweb/htdocs/js/pan/policies
1M /var/appweb/htdocs/js/pan/monitor
1M /var/appweb/htdocs/js/pan/mainui
1M /var/appweb/htdocs/PAN_help/en/wwhelp
1M /var/appweb/htdocs/scripts/cache/3rdparty-noncompressed.javascript
1M /var/appweb/htdocs/js/pan/appframework
1M /var/appweb/htdocs/js/lib/jspdf.umd.min.js.map
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhdata
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhdata
1M /var/appweb/htdocs/PAN_help/en/images
1M /var/appweb/htdocs/PAN_help/es/wwhelp
1M /var/appweb/htdocs/PAN_help/fr/wwhelp
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhelp
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhelp
1M /var/appweb/htdocs/PAN_help/ja/wwhelp
1M /var/appweb/sslvpndocs/global-protect/portal/webfonts
1M /var/appweb/htdocs/js/lib/pan-extjs3-main-ext.js
1M /var/appweb/htdocs/scripts/cache/network.javascript
1M /var/appweb/htdocs/php/locale/ja.php
1M /var/appweb/htdocs/php/locale/fr.php
1M /var/appweb/htdocs/php/locale/es.php
1M /var/appweb/htdocs/js/lib/highcharts
1M or less: 2779 files

/var depth 7 ====================================================

25M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts
9M /var/appweb/htdocs/vendor/setasign/tfpdf/font
8M /var/appweb/htdocs/PAN_help/en/wwhdata/js
5M /var/appweb/htdocs/js/3rdParty/ext/resources
4M /var/appweb/htdocs/js/lib/extjs3/ext
4M /var/appweb/htdocs/PAN_help/fr/wwhdata/js
4M /var/appweb/htdocs/PAN_help/es/wwhdata/js
4M /var/appweb/htdocs/vendor/mongodb/mongodb/tests
3M /var/appweb/htdocs/PAN_help/ja/wwhdata/js
1M /var/appweb/htdocs/php/misc/namedschema/PA-220R-ZTP,multi-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220,multi-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220-ZTP,multi-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220R,multi-vsys-mode.javascript
1M /var/appweb/htdocs/js/pan/device/setup
1M /var/appweb/htdocs/js/3rdParty/CodeMirror/mode
1M /var/appweb/htdocs/php/misc/namedschema/PA-220R-ZTP,single-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220-ZTP,single-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220,single-vsys-mode.javascript
1M /var/appweb/htdocs/php/misc/namedschema/PA-220R,single-vsys-mode.javascript
1M /var/appweb/htdocs/PAN_help/en/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhdata/js
1M /var/appweb/htdocs/vendor/mongodb/mongodb/docs
1M /var/appweb/htdocs/PAN_help/es/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/fr/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/ja/wwhelp/wwhimpl
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhdata/js
1M /var/appweb/htdocs/js/pan/locale/ja.js
1M /var/appweb/htdocs/js/pan/locale/fr.js
1M /var/appweb/htdocs/js/pan/locale/es.js
1M or less: 3251 files

/var depth 8 ====================================================

9M /var/appweb/htdocs/vendor/setasign/tfpdf/font/unifont
8M /var/appweb/htdocs/PAN_help/en/wwhdata/js/search
4M /var/appweb/htdocs/PAN_help/fr/wwhdata/js/search
4M /var/appweb/htdocs/PAN_help/es/wwhdata/js/search
4M /var/appweb/htdocs/js/lib/extjs3/ext/resources
3M /var/appweb/htdocs/js/3rdParty/ext/resources/images
3M /var/appweb/htdocs/PAN_help/ja/wwhdata/js/search
2M /var/appweb/htdocs/vendor/mongodb/mongodb/tests/SpecTests
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/freeserif.z
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/dejavu-fonts-ttf-2.34
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/cid0kr.php
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/cid0cs.php
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/cid0ct.php
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/cid0jp.php
1M /var/appweb/htdocs/js/3rdParty/ext/resources/css
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhdata/js/search
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhdata/js/search
1M or less: 3070 files

/var depth 9 ====================================================

7M /var/appweb/htdocs/PAN_help/en/wwhdata/js/search/pairs
3M /var/appweb/htdocs/js/lib/extjs3/ext/resources/images
3M /var/appweb/htdocs/PAN_help/fr/wwhdata/js/search/pairs
3M /var/appweb/htdocs/PAN_help/es/wwhdata/js/search/pairs
3M /var/appweb/htdocs/PAN_help/ja/wwhdata/js/search/pairs
1M /var/appweb/htdocs/php/3rdParty/tcpdf/fonts/dejavu-fonts-ttf-2.34/status.txt
1M /var/appweb/htdocs/PAN_help/zh_CN/wwhdata/js/search/pairs
1M /var/appweb/htdocs/PAN_help/zh_TW/wwhdata/js/search/pairs
1M /var/appweb/htdocs/js/3rdParty/ext/resources/images/default
1M /var/appweb/htdocs/js/3rdParty/ext/resources/images/yourtheme
1M or less: 1601 files

/var depth 10 ====================================================

1M /var/appweb/htdocs/js/lib/extjs3/ext/resources/images/default
1M /var/appweb/htdocs/js/lib/extjs3/ext/resources/images/yourtheme
1M or less: 4480 files

/var depth 11 ====================================================

1M or less: 953 files

/var depth 12 ====================================================

1M or less: 762 files

/tmp depth 1 ====================================================

17M /tmp

/tmp depth 2 ====================================================

8M /tmp/stats
4M /tmp/modified-schema.xml
1M /tmp/modified-opschema.xml
1M /tmp/ds_appstat.1.prev
1M /tmp/capath
1M or less: 86 files

/tmp depth 3 ====================================================

8M /tmp/stats/pkg
1M or less: 382 files

/tmp depth 4 ====================================================

8M /tmp/stats/pkg/diags

/tmp depth 5 ====================================================

6M /tmp/stats/pkg/diags/devsrv.log
1M /tmp/stats/pkg/diags/paninstaller_content.log
1M or less: 1 files

[akuzhuppilly]

This issue was resolved in 10.2.4 - PAN-206243

https://docs.paloaltonetworks.com/pan-os/10-2/pan-os-release-notes/pan-os-10-2-4-known-and-addressed...

[Raul_Pantaleon]

Hello we have  the same problem in 10.2.4,  I applied the KB High Disk Space Usage on / root partition and How To Clear - Knowledge Base - Palo Alto Networks

but it return this message: 

Server error : op command for client dagger timed out as client is not available

.The last part of KB  importan notes said:

If your FW is a PA-220 running PAN-OS version 10.1.x but less than 10.1.9 then upgrade your device to 10.1.9-h3 or higher to fix issue mentioned in PAN-193452 regarding firewall reaching the maximum disk usage capacity repeatedly in one day. 
If your FW is a PA-220 running PAN-OS version 10.2.x this issue is still under investigation reach out to support if the steps listed above don't help you solve this problem.

The TAC solutions is   Downgrade to 10.1.10 or stiil waiting next release 10.2.5 to fix the issue

At momento is not a option dwoangrade because we are usign SDWAN plugin 3.0.3 

[PavelK]

It looks like that PAN-OS 10.2.5 has addressed both issues mentioned in this thread:

PAN-219659
Fixed an issue where root partition frequently filled up and the following error message was displayed: Disk usage for / exceeds limit, xx percent in use, cleaning filesystem.

PAN-212848
Fixed an issue where attempting to change the disk-usage cleanup threshold to 90 resulted in the error message Server error : op command for client dagger timed out as client is not available.

Kind Regards

Pavel

[JavierChueca]

Hello team:

Upgrading a pa220 from 10.1.10-h2 to 10.2.5, similar issue happening but without disk space error:

In webUi:
~
Failed to install 10.2.5 with the following errors.
SW version is 10.2.5
Installing [content ] into partition "pancfg"
Nothing to install into /mnt/swm/pancfg/...
Installing [panos ] into partition "sysroot0"
Installing packages into /mnt/swm/sysroot0/...
~ 

In the swm log:

08/23/23 15:43:09 swm.pkg ERROR: installing package tzdata-2023c-1.0.1087.pan.noarch needs 604KB on the / filesystem
08/23/23 15:43:09 swm ERROR: Package Installation Error
08/23/23 15:43:09 swm DEBUG: dosys: ['/bin/umount', '/mnt/swm/sysroot0']
08/23/23 15:43:16 swm DEBUG: dosys: ['/bin/umount', '/mnt/swm/pancfg']

 And there is enough disk space available... 

And what is weird is that 10.2.4-h4 was installed without issues...

~> debug swm status

Partition State Version
--------------------------------------------------------------------------------
sysroot0 PENDING-CHANGE 10.2.4-h4
sysroot1 RUNNING-ACTIVE 10.1.10-h2
maint EMPTY None

Regards.

[jasonwald]

Try rebooting first, then upgrade code. Whenever I reboot it clears up a bunch of disk space.

[jasonwald]

Well 10.2.5 looks promising for the PA-220. I haven't seen this much space on root in a long time!

user@XX-XXX-FW> show system disk-space

Filesystem Size Used Avail Use% Mounted on
/dev/root 3.8G 3.1G 512M 87% /

[pbezouska]

Had no problems with disk usage. After upgrading to 10.2.5, I need to wipe panlogs everyday or Panorama becomes unusable due to growing wt collection in pancfg mongoDB instance. Sometimes it overwhelms the unit so much that it won't respond on CLI. And it doesn't even failover. Nice!

[rpanini]

We had this issue on several PA220, after following the KB , you have to reboot the device to cleanup the space.

After >debug software disk-usage aggressive-cleaning enable , you reboot and then you've 90-91% usage of the disk.

The upgrade to 10.2.4-h4 or 10.2.5

[jasonwald]

Mine made is 17 days on 10.2.5 and then today boom 98% full. I would say this is not fixed. So annoying.

[rpanini]

Mine are all between 92 and 94 %