PA 2000 platforms rebooting in our network

[nmsecteam]

We have deployed around 10 pairs of PA 2000 platforms in different networks within our environment.

These networks almost generate the same type of traffic. What we experience is that, these firewalls which ever is active, goes in for an automatic reboot. The traffic is not interrupted bcs of the passive device taking over.

We see this in all the pairs. There are no patterns in the time of the reboot.

Anybody have heard/experienced any similar problems??

[panos]

what is panos version ?

[nmsecteam]

PA 4.1.8 - h3.

We have opened TAC cases. They suggested replacing HDD, which we did.

Even that did not help.Still  experienced it

[panos]

if you see issue with all pairs you should try to upgrade your panos first.(if you did not try - first 4.1.10 then if issue occurs again 5.0.3 )There are some bugs which were not listed on release notes that I saw on some environments.

Chaning hdd is not a solution I think, You are talking about 10pairs

[nmsecteam]

PA TAC had escalated this to their highest level in PA. They had advised this solution of changing hard disks.

From the console logs, we see that the hard disks were freezing.

And  they have not recommended an upgrade yet, bcs they did not see anything from the logs    

[panos]

very strange, how could all disks have this problem ?

upgrading panos is revertable, nothing comes to my mind except for that now

[mikand]

I would recommend you to try 4.1.11 and see if that resolves anything (at least on one pair).

Previous versions of 4.1 series have had some issues (like high mgmtplane cpu usage if I recall correctly) which is supposed to be fixed in 4.1.11 (or was this only for the PANOS 5.x series where 5.0.3 fixed this)?

Also check stuff like temperature and moisture in the area.

If all 10 pairs were delivered at the same time it could be just some bad batch of harddrives if you are unlucky?

But it could also be a case (specially if all 10 pairs were delivered at same time) that the drives have had too many G-forces applied to them during transport (like if the whole container was dropped from above 1 meter of height or such) or something else that happend during transport.

Also if possible would be to install 5.0.3 on one pair and then perform a factory reset (in case something odd happend to the contents of the drives or such).

[ericgearhart]

mikand - the high management plane CPU issue that is fresh in my mind is from the use of the User-ID agent, and that fix is in 4.1.12 (not out yet -

we're upgrading to 4.1.12 as soon as it gets GA released)

[mikand]

Ahh yes... I thought something was off there since 4.1.11 came out first week in february and the high mgmtplane cpu thingy has been around in here since then. First resolution seems to be in 5.0.3 released last week.

[ericgearhart]

Just FYI support assures me that 4.1.12 will contain the high CPU fix as well... it's due out the last week of April (straight from support)

[EdwinD]

For what it is worth, our 2050 HA Pair reboot quite often using PanOS 4.1.10.   I'm currently reviewing the forums for issues with 4.1.12 prior to upgrading.

[JoshGilliland]

Although our 2050's are on version 5.0.4, this sounds like very similar behavior to what we experienced and what we were told would be fixed in 5.0.5. The bug id is 46721.

[rsouthern]

All, just to clarify, BugID 46721 is for a dataplane (DP) restart while doing SSL decryption. This crash will have a backtrace, if there is no backtrace you may be experiencing a different issue. This is a problem specific to 5.0.x code, it does not happen in 4.1.x or 4.0.x. It is triggered by the presence of reordered packets carrying the SSL records. This typically happens with a high volume of traffic. It is fixed in 5.0.5