Disk usage exceeding limit

[mobradovic]

Good Day

I'm getting a system log critical error of - 'Disk usage exceeding limit', 100 percent in use, cleaning filesystem.

Has anyone seen this? and if so how do I fix it.

M

[mobradovic]

There is an large tech support file, which maybe causing the issue. Does anyone know how to delete the file off the firewall?

[HULK]

Hello Mobradovic,

The message that you were receiving " SYSTEM ALERT : critical : "Disk usage exceeds the limit, 100 percent in use, cleaning filesystem" is normal as it is you would have set the alarms for the system logs.

Everytime the logs reaches a certain threshold an alarm will be generated. The big tech-support file is not causing the issue here. Bythe way, a user can not delete a tech-support file from the firewall CLI and GUI. Palo Alto TAC engineer should have access to the root to delete the tech-support file from /tmp directory.

FYI:

Thanks

[mario11584]

I'm getting the same error (but mine is 96%). I don't have any alarms enabled. Just the default "Traffic Log DB 90" settings. This is actually a big deal for us because it seems like when we get these errors our PA-200 data plane restarts (not every time but occassionally).

show system disk-space

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda3             1.9G  1.8G   91M  96% /

/dev/sda5             6.6G  4.2G  2.2G  67% /opt/pancfg

/dev/sda6             1.9G  1.1G  733M  61% /opt/panrepo

tmpfs                 1.3G   67M  1.2G   6% /dev/shm

/dev/sda8             2.4G  715M  1.6G  32% /opt/panlogs

I've had this problem before with another PA-200 we have in a different remote office. Palo Alto had to get root access to the firewall and delete some temp files. Haven't had problems since on that firewall. I've opened a ticket with Palo Alto. I'll post our findings.

[HULK]

Hello Mario,

Ideally, the PAN firewall will automatically purge old logs while it will reach the 100% of it's allocated space on the HDD. Below mentioned doc will explain you about the functionality:

What does alarm "Log Database Exceeds Alarm Threshold Value" mean?

How to Determine How Much Disk Space is Allocated to Logs

When are Logs Purged on the Palo Alto Networks Devices?

If the PAN firewall's data-plane restarted it's own, then you can verify SYSTEM logs just before the incident happened. It might be not related to database purging. Else open a case with support to get an RCA.

Hope this helps.

Thanks

[ymiyashita]

What's the PAN-OS version? Does the error happen only when you try to generate a tech support file? Some old versions (4.1.9 or prior) have debug code enabled and it might be the reason why your box has large tech support file. If you're using new versions, please forget about this.

- Yasu

[mobradovic]

Hi Hulk

Well, that is for sure the PAN will not let you get to root they have to do it, that is an issue for this is a closed network. That being said, I now can't get to the GUI I get "session timed out", looks like I'm going to have to RMA the firewall

M

[mario11584]

I opened up a ticket with support. We weren't able to figure out the problem, even after escalating it to development. This could be partly my fault. Disk-usage was approaching 100% and I didn't want to wait and see what happened when it got there. I ran into this problem before on another PA-200 and we updated the firmware, which fixed the problem. I performed an emergency firmware update which fixed the issue on this PA. Disk usage fell to 75%. I haven't had any problems since.

Regarding my ticket, I was told there wasn't any further analysis that could be done since I did the upgrade, but it's likely the problem will occur again. I guess I'll just keep monitoring the firewall and wait for it to happen again. Also, when they got root access to the 200 they deleted several large files from this partition and it didn't affect the disk-usage, it stayed the same, in fact at one point after purging all the old version software not in use, the disk space increased. This is really odd behavior.

This really seems like a bug. It sounds like the PA is designed to purge files when the disk usage threshold is met but it's apparent that this is NOT happening as it should be. I have not had this problem on our 500s or our 3020s.

I hope this information helps somebody.