Attempt accessing the active and/or passive firewalls fails with the error "fork failed: No space left on device"

[A_Adamski]

Dear and valuable Live Community Members,

I'm wondering if anyone has an issue when trying to access the MGMT interface off the active and/or passive firewalls and getting the error "fork failed: No space left on device"

We were never able to access the firewall and in the end, we rebooted both devices to be able to access the management interface.

We couldn't take the TSF during the crash management, and there was no way to connect to the firewalls during the issue (web, ssh, or console connection):
- via HTTP getting "Critical processes (configd) are down. Please try again later."
- via SSH getting "kex_exchange_identification: read: Connection reset by peer"
- via CONSOLE getting "-- admin: fork failed: No space left on device"

HA member 1

----------------------------------------------------------
> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3   21G  4.8G   15G  25% /
none            7.8G   64K  7.8G   1% /dev
/dev/mmcblk0p5   32G  5.0G   25G  17% /opt/pancfg
/dev/mmcblk0p6   18G  1.5G   16G   9% /opt/panrepo
tmpfs           7.8G  4.2G  3.6G  54% /dev/shm
cgroup_root     7.8G     0  7.8G   0% /cgroup
/dev/mmcblk0p8   22G   12G  9.3G  55% /opt/panlogs
tmpfs            12M   36K   12M   1% /opt/pancfg/mgmt/ssl/private

HA member 2

----------------------------------------------------------
> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3   21G  4.8G   15G  25% /
none            7.8G   64K  7.8G   1% /dev
/dev/mmcblk0p5   32G  5.5G   25G  19% /opt/pancfg
/dev/mmcblk0p6   18G  2.0G   15G  12% /opt/panrepo
tmpfs           7.8G  4.1G  3.7G  53% /dev/shm
cgroup_root     7.8G     0  7.8G   0% /cgroup
/dev/mmcblk0p8   22G   16G  5.1G  76% /opt/panlogs
tmpfs            12M   36K   12M   1% /opt/pancfg/mgmt/ssl/private

After rebooting each one of them, everything works properly, we've logged into the firewalls but no trace of the problem was found (enough free disk was available in all filesystems). 

We have been monitoring but we have not found any answer to what has happened that both firewalls apparently failed at the same time. 

Did anyone had faced similar issues, and maybe know why it happened (what triggered this)?

Is there any way we could protect our firewalls from this issue to not reoccurring in the future?

I would like to ask you for your help and advice on this one.

Thank you in advance!

Cheers!

[Thomas5iron]

I just ran into this same type of issue last night, except it was with two HA Pairs. We had Palo Support engaged and since we had to pull the plug on the firewall, they couldn't review the past logs. 

When looking at our system logs, we can see that a HA Sync failed for both different pairs of firewalls at the same time. When trying to sync a dynamic update. ( Odd I know ) If you go back into your system logs do you have that same error? The event was config-failure. 

Also what OS is your firewall? and What model? We have Two sets up 820's on 10.1.6-h6. 

[OtakarKlier]

Hello,

I agree with opening a support case.

Regards,

[Thomas5iron]

Another update, this has happened again to another 820 Firewall on 10.1.6-h6 that I managed. I am engaged with Palo Support and will keep this thread updated. 

[OtakarKlier]

Hello,

One thing you can try is to delete all code versions you have downloaded except for the one you are running and its base, eg 10.1.6-h6 and 10.1.0. Also to be honest I am still running the base 10.1.6 since the higher versions dont seem to be all stable (just my observation, also not sure if you needed to upgrade to a hotfix, I have had to in the past).

Regards,