PA-3020 - SYSTEM ALERT : critical : Disk usage for / exceeds limit, 95 percent in use

[VishnuPS]

Hi Team,

We are frequently experiencing Disk Usage exceeds limit alerts for PA-3020. s/w version: 9.0.11

Please find our findings below,

Disk Space
==============================

Filesystem Size Used Avail Use% Mounted on
/dev/root 3.8G 3.3G 348M 91% /
none 1.9G 64K 1.9G 1% /dev
/dev/sda5 7.6G 4.6G 2.6G 64% /opt/pancfg
/dev/sda6 3.8G 1.6G 2.1G 44% /opt/panrepo
tmpfs 1.9G 247M 1.6G 14% /dev/shm
/dev/sda8 90G 69G 17G 82% /opt/panlogs

Disk usage
======================================

traffic: Logs and Indexes: 25G Current Retention: 3 days
threat: Logs and Indexes: 14G Current Retention: 10 days
system: Logs and Indexes: 3.3G Current Retention: 184 days
config: Logs and Indexes: 3.3G Current Retention: 1695 days
alarm: Logs and Indexes: 48K Current Retention: 0 days
trsum: Logs and Indexes: 5.7G Current Retention: 27 days

Quotas
=======================================

system: 4.00%, 3.236 GB Expiration-period: 0 days
config: 4.00%, 3.236 GB Expiration-period: 0 days
alarm: 3.00%, 2.427 GB Expiration-period: 0 days
appstat: 4.00%, 3.236 GB Expiration-period: 0 days

/tmp/cli/logs/show_log_system.txt
==================================

2021-05-08 14:26:13 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-14 10:41:53 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-16 12:12:08 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-18 16:45:22 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-19 17:18:28 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-19 22:00:37 Issue Warning MP Disk Disk Space /dev/root is 91% utilized

2021-05-21 16:27:41 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-23 10:45:54 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-25 06:01:07 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-27 06:01:21 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

2021-05-27 13:16:22 Issue Notice MP Disk / Disk usage exceeds limit. Attempting to clean filesystem.

MP LOGS:
============
2021-05-27 13:16:22.906 +0530 --- filesystem mp-monitor.log

2021-05-27 13:16:22 Disk usage for / exceeds limit, 95 percent in use, cleaning filesystem

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/rasmgr.log.old, size 10485828

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/ikemgr.log.old, size 10485842

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/devsrv.log.old, size 10485854

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/dnsproxyd.log.old, size 10485794

mp-monitor.log 2021-05-27 13:16:22Removing /var/log/pan/authd.log.old, size 10485777

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/qtrace_routed.log.1, size 10485842

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/paninstaller_content.log.old, size 10550487

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/appweb3-sslvpn.log.old, size 10491140

mp-monitor.log 2021-05-27 13:16:22 Removing /var/log/pan/logrcvr.log.old, size 10485888

mp-monitor.log 2021-05-27 13:16:22 Removing /opt/dpfs/var/log/pan/panio.log.old, size 10485816

2021-05-19 21:49:24 2021-05-19 21:49:24.990 +0530 ### MS-DB: RuleHit update: /opt/pancfg/mgmt/devices/localhost.localdomain/rule-hit-count-db.txt
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 9 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 9 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!
mp ms.log.old 2021-05-19 21:52:05 2021-05-19 21:52:05.934 +0530 Warning: pan_hash_init(pan_hash.c:113): nbuckets 2000 is not power of 2!

How to resolve this disk space issue and guide us if we need to take any TAC support from PA.

[MP18]

@VishnuPS 

Please follow below link to delete the old logs from MP plane

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRXCA0

Regards

[OtakarKlier]

Hello,

Also the PAN will remove old logs to make room for newer ones. Think of it as a rolling log file.

Regards,