topic Re: PA-3050 stops processing traffic in General Topics

PA-3050 stops processing traffic

mjcorriganaaa — Mon, 16 Dec 2013 21:55:21 GMT

Has anyone had a PA-3050 stop processing traffic? Our PA-3050 started dropping all traffic today (internet access, DMZ, etc.), we failed over to the standby unit and were able to restore service.

Currently we have a support ticket opened but wanted to know if anyone here has had a similar experience. Thanks!

Re: PA-3050 stops processing traffic

jdub01 — Mon, 06 Jan 2014 21:54:49 GMT

We are having similar issue. 3 times in 2 weeks primary 3050, running 5.0.8 stops passing all traffic in v-wire mode and won't fail-over automatically. We forced fail-over to passive box, rebooted fw1, and failed back to it just to have it happen again a few weeks later. We are currently pushing all traffic through the backup 3050 until PAN comes up with recommendation, or fix.

Re: PA-3050 stops processing traffic

arvesynd — Wed, 30 Apr 2014 10:40:44 GMT

Hi,

I've had this happen several times as well, last one today. We also have a HA cluster, and the primary (and active) firewall suddenly stops routing traffic, and the secondary (passive) does not try to take over, and acts like everything is OK.

Had to reboot the primary firewall for the secondary to become active, and I would like to avoid this happening again.

I've sent tech dumps to our support contact, so hopefully I'll get a better answer than "I'm sure this won't happen again" this time. Running 5.0.11, by the way.

Re: PA-3050 stops processing traffic

HULK — Wed, 30 Apr 2014 14:40:33 GMT

Hello As-mg,

Could you please verify PAN brdagent log through CLI command and verify if matching with below mentioned symptoms.

A. PAN> less dp-log brdagent.log

1. Error message like : "need to reset ocelot link as XX error packet seen"

2. link flap messages similar to above.

3. XGE link error.

4. Check system logs for critical logs indicating “DP packet descriptor leak detected on slot 1 dp0” or similar.

Thanks

Re: PA-3050 stops processing traffic

arvesynd — Wed, 30 Apr 2014 20:30:11 GMT

Jumbo frames are not active for our HA links. Forgot to mention that this is a 3020 cluster.

After contacting support their best suggestion was to update to 5.0.12 or 6.0.2.

I have upgraded both to 5.0.12 tonight, but I have a feeling that the bug fixed in 6.0.2 better describes our situation, so I will upgrade the firewalls to 6.0.2 tomorrow.

Fixed in 5.0.12/6.0.1: When High Availability Active/Passive peers lost communication on HA1 and HA2 links, a race condition caused the dataplane to restart.

Fixed in 6.0.2: Fixed an issue with PA-3000 Series devices where traffic could stop passing through the firewall or the dataplane could restart due to an internal path monitoring failure.

Re: PA-3050 stops processing traffic

HULK — Wed, 30 Apr 2014 20:44:56 GMT

Hello As-mg,

It's recommended to run the signature pattern matching (AHO) on software

> debug dataplane fpga set sw_aho yes

Thanks

Re: PA-3050 stops processing traffic

networkadmin — Thu, 01 May 2014 08:19:55 GMT

We had this happen on a 500 running 5.0.10.

At the weekend the box spontaneously rebooted - according to support "They've seen it happen with 5.0.x and have no idea why" - there was nothing useful in the tech support bundle so right now we're just waiting to see if it happens again.

Re: PA-3050 stops processing traffic

marcin_koprowski — Mon, 05 May 2014 07:22:20 GMT

I had the same situation on a 500 with 5.0.11 one week ago. Device restarted suddenly without any reason in logs

Re: PA-3050 stops processing traffic

Marcin — Tue, 06 May 2014 15:01:19 GMT

Apparently we had encountered the same behavior on our 3020 cluster running 6.0.0. The device stopped processing any traffic and there was no evidence of what happened in logs. Restarting dataplane seems to be a working solution for us, yet i'd like to know what was the cause.

I also did verify dp-log brdagent.log as HULK mentioned above and we had these messages:

- Flapping Ocelot link 1

- Error: poll_func(3000/osprey_oct.c:164): Need to reset ocelot link as 51 error packets seen!

Does it mean i should run debug dataplane fpga set sw_aho yes command on our boxes or simply go for 6.0.2?

Re: PA-3050 stops processing traffic

HULK — Tue, 06 May 2014 15:37:58 GMT

Hello Marcin,

I would recommend you to set the pattern matching on software (> debug dataplane fpga set sw_aho yes), instead of PAN OS upgrade.

Thanks

Re: PA-3050 stops processing traffic

arvesynd — Wed, 14 May 2014 10:27:31 GMT

This issue occured again (third time) on our 3020 HA cluster today, running 6.0.2.

Awaiting response from support.

Re: PA-3050 stops processing traffic

HULK — Wed, 14 May 2014 23:14:36 GMT

Hello as-mg,

I would recommend you to set the pattern matching on software (> debug dataplane fpga set sw_aho yes) to avoid any future occurrence.

Thanks

Re: PA-3050 stops processing traffic

arvesynd — Thu, 15 May 2014 04:47:12 GMT

This was the suggestion given us by support as well.

However, they also did say that this change will be lost each time we reboot a firewall, which I would concider less handy and make this a not-so-much as permanent fix.

What we're going to do from now on is also have a computer connected to the failing device via the console, and keep it logging in case the issue occurs again.

It's been under a month since it last happened (PANOS 5.0.11) and the issue still persists on PANOS 6.0.2.

If you have any further suggestions, I would love to hear from you.

Re: PA-3050 stops processing traffic

HULK — Thu, 15 May 2014 05:01:28 GMT

Hello as-mg,

There is a BUG open with engineering to fix this issue. I do agrre, this is not a permanent fix, only things, you need to re-apply the command if you reboot the FW.

Thanks

Re: PA-3050 stops processing traffic

mizhou — Wed, 21 May 2014 07:17:31 GMT

The reboot for PA 500 has been fixed in 5.0.12, you may upgrade to the new version now.