PA-220s randomly crashing

[Claw4609]

We are having a large number of our PA-220s randomly crashing. No critical system logs are see shortly before the crash, the device just goes down and they are logs of dataplane starting up like 30 minutes later. Our other models are fine, its only the 220s we are having issues with. We have had a critical TAC ticket for the last few days and it was initially believed we were hitting a bug in 10.1.8-h2 so we were instructed to upgrade to 10.1.9-h1, that didnt fix our issue, we were then told to upgrade to 10.2.4 but that didn't fix our issue. We're currently communicating with TAC but we're just looking to get this issue resolved.

There were no changes pushed to the Palos shortly before this happened. We have been running 10.1.8-h2 for months with no issues. 

I'm aware this is limited information, but anyone have any ideas.

[MP18]

@Claw4609 

Are the Interfaces in Vwire mode or Layer 3 mode.

We are running Few PA 220 on Version 10.2.5 with no issues so far in Layer 3 mode.

Other than some BUG cannot say much about this  issue.

Try to Escalate with PA to go level 3 and see what they ??

Regards

Mahesh

[Claw4609]

Our interfaces are layer3. This case has been escalated and our SE and Sale Manager have been brought in to push as well. Its odd it just started happening recently. Guess we'll see what TAC can come back with. 

[MP18]

@Claw4609 Recently I have seen BUG issues with PA on firewalls and Panorama.

And to get the Level 3 you need escalation done by your SE.

Regards

Mahesh

[BPry]

@Claw4609,

Haven't seen any abnormal restarts on our PA-220 fleet running 10.1.9-h1 with layer3 interfaces. It'll be interesting to hear what your issue is eventually traced back to, but since it's followed you through a major version upgrade I'm just guessing that it'll be a combination of something specific about your configuration.

I primarily wanted to comment just to say that I would always push back on TAC when they recommend doing a major version bump to address an issue. I'd personally have them show me the ID that they think I'm running into and verify that they've ran the config in their lab without issues on the hardware in question, not a VM series or anything like that. Unless you're running on an unsupported version, I'd personally want some reassurance that the issue is going to be addressed before they attempt to bump me to the next major release that would demand significant validation before I'd even roll it out. Otherwise you run the risk of introducing new issues in addition to what you're already looking to solve. 

[Raido_Rattameister]

When I had this issue with one specific 220 in January with 10.2.3 and 10.2.3-h4 then TAC said issue will be fixed in 10.2.4.

There was no good workaround in any of 10.1.x and 10.2.x releases so I downgraded to 9.1.

Number of other 220s ran 10.2.x fine without rebooting.

I see you are on 10.2.4 already so in case it is same bug then maybe it was pushed further.

[RyanWagoner]

Was there a resolution for this? I'm seeing random reboots on a PA-220 with 10.2.5. No events are logged when this occurs.

[Claw4609]

We actually just ended up replacing all our 220s with 440s that Palo provided us because it was months before they identified the issue and theyre still not sure why it started happening. There was a debug build that supported provided that fixed the issue. 

This is what support said:

The debug build was built based on a potential theory that the issue was caused by a situation where by two cores handle the same work or the same session at the same time.

So they used the debug build to verify if the kernel driver (octeon-pow-ethernet) caused the issue.

[telwood]

I am experiencing the same issue, was there ever a fix recommended other than the upgrade of the hardware? It just randomly started for me 2 days ago, and it is causing a lot of issues.

[MP18]

@telwood 

We have few PA 220 and i am running 10.2.6 without any issues in V-Wire mode.

Regards