topic Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem" in General Topics

opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Tue, 28 Oct 2014 14:08:18 GMT

Hi Guys,

What should i do? who can help me?

Thanks in advance

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

HULK — Tue, 28 Oct 2014 14:14:53 GMT

"Disk usage exceeds limit, 98 percent in use, cleaning filesystem" ------------

Reference DOC:

How to Determine How Much Disk Space is Allocated to Logs

How and When to Clear Disk Space on the Palo Alto Networks Device

When are Logs Purged on the Palo Alto Networks Devices?

What does alarm "Log Database Exceeds Alarm Threshold Value" mean?

Hope this helps.

Thanks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

HULK — Tue, 28 Oct 2014 14:20:10 GMT

What PAN OS version is running on your appliance...?

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

HULK — Tue, 28 Oct 2014 14:22:56 GMT

Please share the CLI output from your PAN firewall:

admin@DADA> show system disk-space

Thanks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Tue, 28 Oct 2014 14:26:30 GMT

Hi Tigran,

From log it seems firewall is doing its job and purging old logs.

Please provide us output for "show system disk-space". If any of the partition is still 100% utilized than its an issue.

In that case call TAC someone will help to clear space.

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Tue, 28 Oct 2014 14:36:01 GMT

Guys,

Thanks for quick response.

Please

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Tue, 28 Oct 2014 14:37:46 GMT

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 1.9G 1.8G 44M 98% /

/dev/sda5 6.6G 3.7G 2.6G 60% /opt/pancfg

/dev/sda6 1.9G 887M 946M 49% /opt/panrepo

tmpfs 1.3G 67M 1.2G 6% /dev/shm

/dev/sda8 2.4G 1.9G 392M 83% /opt/panlogs

Here are the details, the root directory is full 98%, i was trying to use ls command like in linux/unix, but without any luck.How can i see what do i have in that dircetory?

Thanks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

HULK — Tue, 28 Oct 2014 14:59:31 GMT

Hello Tigran,,

It seems the file system /dev/sda3 is almost full and only 44MB space is remaining on the HDD. You may contact with PAN support to delete few unwanted files from root#.

I have seen a similar problem in the past, where PAN FW is logging failed SSH login attempts (/var/log/btmp ). This file is not rotated and can grow and consume all the space on HDD. This issue is most likely to occur on devices like PA-200.

Bug 66208 - SSH bruteforce attack on firewall interface caused the /var/log/btmp to get huge causing system disk space full

Fixed: PAN OS 6.1.0/6.0.7

Hope this helps.

Thanks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Tue, 28 Oct 2014 17:18:10 GMT

Hi Tigran,

I think its PA-200 box. It has a known issue where root patition is almost full "/dev/sda3 1.9G 1.8G 44M 98% /".

I would suggest you to contact TAC. They will removt btmp file and it should be fine after that. Let me know if that helps.

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

Retired Member — Tue, 28 Oct 2014 17:54:54 GMT

we had the same issue on many pa200 fw

any solution without opening a case ?

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Tue, 28 Oct 2014 19:08:33 GMT

Hi PANOS,.

End user can not do anything as it requires root access to delete files.

Its a bug/66208, its fixed in 6.0.7. So upgrade is the only solution.

6.0.7 will be released soon

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

HULK — Tue, 28 Oct 2014 19:21:38 GMT

@ Hardik FYI- This Bug is also fixed in 6.1.0, which has been released yesterday. So, if any of the PAN customer is having this issue, they may upgrade it to 6.1.0.

Thanks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

_slv_ — Tue, 28 Oct 2014 19:41:36 GMT

Hello

I have also problem with PA200, recently twice time I had to reboot this device because it stopped passing traffic. During reboot proces (that took about 15minuts!!!) I got email with subject "SYSTEM ALERT : critical : "Disk usage exceeds limit, 99 percent in use, cleaning filesystem""

Also I realized that during commit I got info that I havent antivirus update ?!???!

admin@PA-200> show system disk-space

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 1.9G 1.6G 294M 84% /

/dev/sda5 6.6G 3.7G 2.7G 58% /opt/pancfg

/dev/sda6 1.9G 885M 948M 49% /opt/panrepo

tmpfs 1.3G 67M 1.2G 6% /dev/shm

/dev/sda8 2.4G 2.0G 339M 86% /opt/panlogs

I think that my problem is free space on /dev/sda8 not on root partition.

Tommorow morning I will have remote session with TAC, we will see how much space he get.

@ Hardik - are You sure that 6.0.7 will be releases soon? 6.0.6 was released tw days ago so I think that we must wait another 2 months for it.

Has anyone similiar to my problems with PA200?

Regards

SLawek

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

Retired Member — Tue, 28 Oct 2014 19:47:44 GMT

Thanks HULK for information.

upgrading to 6.1 will be good.

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Tue, 28 Oct 2014 20:16:37 GMT

Guys,

We connected with their support.They are looking at the problem, I'm waiting for their solution.

Best

Tigran

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Tue, 28 Oct 2014 20:17:37 GMT

Hi SLV,

Yes you are right, 6.0.6 has some time. As suggested 6.1.0 has fix.

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Tue, 28 Oct 2014 21:00:24 GMT

Hi Tigran,

What did TAC say ? Did they suggested same bug and solution ?

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Thu, 30 Oct 2014 10:01:36 GMT

Hi hshah,

Here is the answer

Hi Tigran,

Thank you for calling Palo Alto Networks. It was nice talking with you.

----------------------------------------------------------------------------------------

Issue Faced: Storage space on "/" partition was 98%

----------------------------------------------------------------------------------------

Steps Taken:

--I logged in as root and deleted the file in /var/log which was consuming lots of space

--This file logs failed SSH attemps and can increase in size during brute force attack.

--The issue is resolved in PAN-OS 6.1 and is tentatively scheduled to be resolved in PAN-OS 6.0.7 as well:

66208 A brute-force attack on an unprotected management interface on the firewall caused the /var/log/btmp log file to inflate and consume available disk space. With this fix, PAN-OS enables a log rotation function for failed SSH logins, such as those from brute-force attacks.

As discussed I will now proceed with case closure.

Thanks and Regards

Chetan Sharma

Technical Support Engineer

Shift Time : 2:00 PM – 11:00 PM CDT (GMT-5) Monday-Friday

Palo Alto Networks

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

hshah — Thu, 30 Oct 2014 12:42:42 GMT

Hi Tigran,

I am glad is same bug and same resolution. Let us know for additional queries.

Regards,

Hardik Shah

Re: opaque: "Disk usage exceeds limit, 98 percent in use, cleaning filesystem"

TigranGevorgyan — Fri, 31 Oct 2014 09:39:39 GMT

Ok, Thanks Hardik for Support.The Only thing i didn't get, why PAN doesn't give the clients the root accesses to root directory to clean the diskspace themselves?

Best

Tigran