topic Re: HA broken after upgrading to 6.0.3 in General Topics

HA broken after upgrading to 6.0.3

MMCiobanu — Mon, 16 Jun 2014 22:09:56 GMT

Hi,

We have just upgraded our 5020s and 3020s to 6.0.3 and encountered an issue, where the secondary device became the Active one and the primary displays this error, only on the 5020:

does anybody else had this issue or knows how to solve it?

Thank you.

Re: HA broken after upgrading to 6.0.3

kadak — Mon, 16 Jun 2014 22:22:41 GMT

Hello MMCiobanu,

This could be due to the dataplane hitting the kernel NMI causing the control plane to exit with internal path monitoring failure. We need to determine the reason for the initial NMI failure which could be due to either a software caused memory corruption or due to a hardware problem.

That calls for a support case to be opened so that the assigned TAC engineer can investigate the tech support files. Remember, the tech support files needs to be gathered along with crash files (if any) before rebooting the firewall.

Hope that helps!

Thanks and regards,

Kunal Adak

Re: HA broken after upgrading to 6.0.3

hopcio — Mon, 16 Jun 2014 22:50:13 GMT

Hi,

Got the same error after upgrade a pair of PA-5020 in HA A/P and the primary went to maint mode.

Do you have the following error by any chance? "control_plane: Depend script failed max times".

I'm opening a case also, but if you have any update a appreciate any feedback

Re: HA broken after upgrading to 6.0.3

MMCiobanu — Tue, 17 Jun 2014 14:42:30 GMT

Hi hopcio,

We did not have the error you had; our device went into "non-functional" mode. We had a case open, and sent the tech-support file, but we did not hear back by the end of the day and decided to schedule a reboot of the broken unit, while the second one was still Active and passing traffic. The reboot fixed the issue, so far. But, still don't know what caused it in the first place.

I will post an update once hearing back from support.

Re: HA broken after upgrading to 6.0.3

Oleksandr — Tue, 17 Jun 2014 16:22:37 GMT

Hi!

Any updates for this?

Thanks,

Alex

Re: HA broken after upgrading to 6.0.3

MMCiobanu — Tue, 17 Jun 2014 16:28:49 GMT

No update from the PaloAlto support team; but, as I said, we solved the issue after rebooting the broken unit. We still don't know what caused the issue in the first place.

Re: HA broken after upgrading to 6.0.3

pulukas — Sat, 21 Jun 2014 12:19:50 GMT

Thanks for sharing and keep us posted on the progress here.

We are looking at when to move from 5 to 6 and are running four A/A HA clusters in the DC. I'm glad the fix is a reboot, but wondering what may bring the issue back since the root cause is not identified yet.

Re: HA broken after upgrading to 6.0.3

MMCiobanu — Sat, 21 Jun 2014 15:22:04 GMT

PaloAlto still investigating, this is the response:

"From the analysis of the collected the dataplane restart happened initially at 17:02 on 06/15 where an internal path monitoring failure detected . The internal path monitoring failure occurs when internal monitor packets between the daemons in the firewall are not ACKed timely. When the packets are not ACKed, the firewalls assumes some daemon is down and triggers a dataplane restart.

2014/06/15 16:56:45 high ha state-c 0 HA Group 30: Moved from state Passive to state Active
2014/06/15 17:02:39 high general general 0 9: path_monitor HB failures seen, triggering HA DP down
2014/06/15 17:02:39 critical ha datapla 0 HA Group 30: Dataplane is down: path monitor failure
2014/06/15 17:02:39 critical ha state-c 0 HA Group 30: Moved from state Active to state Non-Functional
2014/06/15 17:02:53 high general general 0 flow_mgmt: exiting because missed too many heartbeats
2014/06/15 17:03:17 critical general general 0 Internal packet path monitoring failure, restarting dataplane

Device became active after the restart of dataplane and again detected the internal path monitoring failure and triggered the data-plane restart . This time dataplane didn't restart and was down causing the device to be in non-functional state until you rebooted the device next day

2014/06/15 17:07:30 info ha state-c 0 HA Group 30: Moved from state Initial to state Passive
2014/06/15 17:09:42 high ha state-c 0 HA Group 30: Moved from state Passive to state Active
2014/06/15 17:33:46 high general general 0 9: path_monitor HB failures seen, triggering HA DP down
2014/06/15 17:33:46 critical ha datapla 0 HA Group 30: Dataplane is down: path monitor failure
2014/06/15 17:33:46 critical ha state-c 0 HA Group 30: Moved from state Active to state Non-Functional
2014/06/15 17:34:25 critical general general 0 Internal packet path monitoring failure, restarting dataplane
014/06/15 22:24:45 high general general 0 flow_mgmt: exiting because missed too many heartbeats
2014/06/15 22:25:05 critical general general 0 Internal packet path monitoring failure, restarting dataplane

The DP1 CPU was under high load for that moment right before the dataplane restart and did see some software buffer depletion which could have triggered the internal path failure. "

They opened a bug with development team to validate the findings and analyze the crash files and logs further. And they want us to call them if this happens again, so they can do a live debug.

Re: HA broken after upgrading to 6.0.3

pulukas — Sat, 21 Jun 2014 17:01:23 GMT

Thanks for the details.

I would be concerned about finding the root cause on this one. A series of events that brings down a cluster member is a pretty serious bug. But if it only occurs immediately after the upgrade and clears with a reboot it possibly could be covered in the upgrade outage window.

Did you upgrade and reboot both members together or one at a time? If separately primary or secondary first?

Re: HA broken after upgrading to 6.0.3

MMCiobanu — Wed, 30 Jul 2014 16:33:04 GMT

This happened again after a few months now since we have upgraded;

It happened after a reboot - secondary device was booted first, then primary, then secondary again. After every reboot, I have waited a few minutes and made sure HA was fine; however, in the morning the secondary device became the primary and displaying (Dataplane down: path monitor failure), and the primary became non functional.

After restarting the dataplane on the non functional device, it came back as passive.

From the logs it shows that it kept restarting the dataplane for a few hours.

We have contacted PaloAlto support again and they are trying to figure out what causes these issues after the restart.

Re: HA broken after upgrading to 6.0.3

pulukas — Wed, 30 Jul 2014 21:07:52 GMT

Thanks for keeping us posted.

Your reboot procedure is the same one we using in upgrades of a cluster, one at a time with the secondary first.

This is a serious bug. Do I understand you correctly that traffic is still passing with the secondary node as primary? Or does this cause an full network outage?

Re: HA broken after upgrading to 6.0.3

MMCiobanu — Wed, 30 Jul 2014 21:14:43 GMT

hi Steven Puluka

That's right, traffic is still passing through the secondary node as it became primary, actually it became the only node functional. There was not a network outage.

we got the failed node back in the HA by restarting the dataplane on it: "request restart dataplane". This made it re-join the cluster as the passive node.

Now, waiting for PaloAlto to come back with some kind of resolution.

Re: HA broken after upgrading to 6.0.3

j.liu — Wed, 30 Jul 2014 21:39:07 GMT

Do you mind sharing the case number? I am curious of details. I have a client in the process of upgrading to 6.0.3,

Re: HA broken after upgrading to 6.0.3

LuisLorienteAra — Wed, 13 Aug 2014 10:43:43 GMT

We are suffering the same issue.

Now rebooting secondary node.

Our soft version is 6.0.2 in both nodes.

Re: HA broken after upgrading to 6.0.3

Satish — Wed, 13 Aug 2014 11:36:44 GMT

Hi Luis,

What error are showing in logs file.

PAN-OS 6.0.4: Addressed Issues

Some issue are addressed in pan OS 6.0.4.

Regards

SAtish

Re: HA broken after upgrading to 6.0.3

LuisLorienteAra — Wed, 13 Aug 2014 11:39:14 GMT

Re: HA broken after upgrading to 6.0.3

Satish — Wed, 13 Aug 2014 11:45:16 GMT

Hi MMC,

64591 (The fix in this bug resolves two issues: o PA-3000 Series firewalls could stop passing traffic due to errors in the internal datapath. A registry value was changed in the dataplane packet processor to prevent this error. o When highly compressed traffic is passing through PA-7050, PA-5000 Series, or PA-3000 Series firewalls, this caused some additional latency in other traffic passing through the firewall. This fix prevents the latency from being introduced to the traffic by changing the amount of data being handled by the signature matching FPGA.)

Can you try to upgrade one more PAN OS 6.0.3 to PAN OS 6.0.4. i hope its helpful for you. thanks

Regards

Satish

Re: HA broken after upgrading to 6.0.3

LuisLorienteAra — Wed, 13 Aug 2014 11:49:10 GMT

Hi Satish, thank you for your response, I'm reading 6.0.4 release notes and we'll upgrade to 6.0.4 as soon as possible.

thanks again.

Re: HA broken after upgrading to 6.0.3

Satish — Wed, 13 Aug 2014 12:00:22 GMT

Your welcome Luis,

Let me know if you are facing any other challenge.

Regards

Satish

Re: HA broken after upgrading to 6.0.3

HULK — Wed, 13 Aug 2014 19:36:30 GMT

Hello Luis,

For the time being, you may apply the below mentioned command as a workaround:

admin@PA-3020> set system setting packet-path-test enable no

Thanks