topic Re: Disk usage exceeding limit in General Topics

Disk usage exceeding limit

mobradovic — Thu, 15 May 2014 15:45:29 GMT

Good Day

I'm getting a system log critical error of - 'Disk usage exceeding limit', 100 percent in use, cleaning filesystem.

Has anyone seen this? and if so how do I fix it.

Re: Disk usage exceeding limit

mobradovic — Thu, 15 May 2014 16:18:22 GMT

There is an large tech support file, which maybe causing the issue. Does anyone know how to delete the file off the firewall?

Re: Disk usage exceeding limit

HULK — Fri, 16 May 2014 21:02:20 GMT

Hello Mobradovic,

The message that you were receiving " SYSTEM ALERT : critical : "Disk usage exceeds the limit, 100 percent in use, cleaning filesystem" is normal as it is you would have set the alarms for the system logs.

Everytime the logs reaches a certain threshold an alarm will be generated. The big tech-support file is not causing the issue here. Bythe way, a user can not delete a tech-support file from the firewall CLI and GUI. Palo Alto TAC engineer should have access to the root to delete the tech-support file from /tmp directory.

FYI:

Thanks

Re: Disk usage exceeding limit

mario11584 — Mon, 19 May 2014 21:48:34 GMT

I'm getting the same error (but mine is 96%). I don't have any alarms enabled. Just the default "Traffic Log DB 90" settings. This is actually a big deal for us because it seems like when we get these errors our PA-200 data plane restarts (not every time but occassionally).

show system disk-space

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 1.9G 1.8G 91M 96% /

/dev/sda5 6.6G 4.2G 2.2G 67% /opt/pancfg

/dev/sda6 1.9G 1.1G 733M 61% /opt/panrepo

tmpfs 1.3G 67M 1.2G 6% /dev/shm

/dev/sda8 2.4G 715M 1.6G 32% /opt/panlogs

I've had this problem before with another PA-200 we have in a different remote office. Palo Alto had to get root access to the firewall and delete some temp files. Haven't had problems since on that firewall. I've opened a ticket with Palo Alto. I'll post our findings.

Re: Disk usage exceeding limit

HULK — Mon, 19 May 2014 22:02:14 GMT

Hello Mario,

Ideally, the PAN firewall will automatically purge old logs while it will reach the 100% of it's allocated space on the HDD. Below mentioned doc will explain you about the functionality:

What does alarm "Log Database Exceeds Alarm Threshold Value" mean?

How to Determine How Much Disk Space is Allocated to Logs

When are Logs Purged on the Palo Alto Networks Devices?

If the PAN firewall's data-plane restarted it's own, then you can verify SYSTEM logs just before the incident happened. It might be not related to database purging. Else open a case with support to get an RCA.

Hope this helps.

Thanks

Re: Disk usage exceeding limit

ymiyashita — Tue, 20 May 2014 00:32:55 GMT

What's the PAN-OS version? Does the error happen only when you try to generate a tech support file? Some old versions (4.1.9 or prior) have debug code enabled and it might be the reason why your box has large tech support file. If you're using new versions, please forget about this.

- Yasu

Re: Disk usage exceeding limit

mobradovic — Tue, 03 Jun 2014 13:17:57 GMT

Hi Hulk

Well, that is for sure the PAN will not let you get to root they have to do it, that is an issue for this is a closed network. That being said, I now can't get to the GUI I get "session timed out", looks like I'm going to have to RMA the firewall

Re: Disk usage exceeding limit

mario11584 — Tue, 03 Jun 2014 14:26:20 GMT

I opened up a ticket with support. We weren't able to figure out the problem, even after escalating it to development. This could be partly my fault. Disk-usage was approaching 100% and I didn't want to wait and see what happened when it got there. I ran into this problem before on another PA-200 and we updated the firmware, which fixed the problem. I performed an emergency firmware update which fixed the issue on this PA. Disk usage fell to 75%. I haven't had any problems since.

Regarding my ticket, I was told there wasn't any further analysis that could be done since I did the upgrade, but it's likely the problem will occur again. I guess I'll just keep monitoring the firewall and wait for it to happen again. Also, when they got root access to the 200 they deleted several large files from this partition and it didn't affect the disk-usage, it stayed the same, in fact at one point after purging all the old version software not in use, the disk space increased. This is really odd behavior.

This really seems like a bug. It sounds like the PA is designed to purge files when the disk usage threshold is met but it's apparent that this is NOT happening as it should be. I have not had this problem on our 500s or our 3020s.

I hope this information helps somebody.