Attempt accessing the active and/or passive firewalls fails with the error "fork failed: No space left on device"

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Attempt accessing the active and/or passive firewalls fails with the error "fork failed: No space left on device"

L2 Linker

Dear and valuable Live Community Members,

 

I'm wondering if anyone has an issue when trying to access the MGMT interface off the active and/or passive firewalls and getting the error "fork failed: No space left on device"

 

We were never able to access the firewall and in the end, we rebooted both devices to be able to access the management interface.

We couldn't take the TSF during the crash management, and there was no way to connect to the firewalls during the issue (web, ssh, or console connection):
- via HTTP getting "Critical processes (configd) are down. Please try again later."
- via SSH getting "kex_exchange_identification: read: Connection reset by peer"
- via CONSOLE getting "-- admin: fork failed: No space left on device"

 

HA member 1

----------------------------------------------------------
> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3   21G  4.8G   15G  25% /
none            7.8G   64K  7.8G   1% /dev
/dev/mmcblk0p5   32G  5.0G   25G  17% /opt/pancfg
/dev/mmcblk0p6   18G  1.5G   16G   9% /opt/panrepo
tmpfs           7.8G  4.2G  3.6G  54% /dev/shm
cgroup_root     7.8G     0  7.8G   0% /cgroup
/dev/mmcblk0p8   22G   12G  9.3G  55% /opt/panlogs
tmpfs            12M   36K   12M   1% /opt/pancfg/mgmt/ssl/private

 

HA member 2

----------------------------------------------------------
> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3   21G  4.8G   15G  25% /
none            7.8G   64K  7.8G   1% /dev
/dev/mmcblk0p5   32G  5.5G   25G  19% /opt/pancfg
/dev/mmcblk0p6   18G  2.0G   15G  12% /opt/panrepo
tmpfs           7.8G  4.1G  3.7G  53% /dev/shm
cgroup_root     7.8G     0  7.8G   0% /cgroup
/dev/mmcblk0p8   22G   16G  5.1G  76% /opt/panlogs
tmpfs            12M   36K   12M   1% /opt/pancfg/mgmt/ssl/private

 

After rebooting each one of them, everything works properly, we've logged into the firewalls but no trace of the problem was found (enough free disk was available in all filesystems). 

We have been monitoring but we have not found any answer to what has happened that both firewalls apparently failed at the same time. 

 

Did anyone had faced similar issues, and maybe know why it happened (what triggered this)?

Is there any way we could protect our firewalls from this issue to not reoccurring in the future?

 

I would like to ask you for your help and advice on this one.

 

Thank you in advance!

Cheers!

 

4 REPLIES 4

L1 Bithead

I just ran into this same type of issue last night, except it was with two HA Pairs. We had Palo Support engaged and since we had to pull the plug on the firewall, they couldn't review the past logs. 

When looking at our system logs, we can see that a HA Sync failed for both different pairs of firewalls at the same time. When trying to sync a dynamic update. ( Odd I know ) If you go back into your system logs do you have that same error? The event was config-failure. 

Also what OS is your firewall? and What model? We have Two sets up 820's on 10.1.6-h6. 

Cyber Elite
Cyber Elite

Hello,

I agree with opening a support case.

Regards,

Another update, this has happened again to another 820 Firewall on 10.1.6-h6 that I managed. I am engaged with Palo Support and will keep this thread updated. 

Cyber Elite
Cyber Elite

Hello,

One thing you can try is to delete all code versions you have downloaded except for the one you are running and its base, eg 10.1.6-h6 and 10.1.0. Also to be honest I am still running the base 10.1.6 since the higher versions dont seem to be all stable (just my observation, also not sure if you needed to upgrade to a hotfix, I have had to in the past).

Regards,

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!