Bug: 8.1.7/8.1.8 PA-5200 AUX Ports

[BPry]

Just FYI,

There is currently a bug within 8.1.7 and 8.1.8 that renders the AUX ports inoperable on the PA-5200 series, which depending on your configuration can cause issues. The Bug ID is PAN-105737 and it's been fixed in 9.*, but you might want to hold off upgrading until its been fixed or update your configuration to remove dependency on AUX interfaces.  

[Pzilla]

Thanks for the heads-up BPry. Looking at the PAN-OS 8.1.8 release notes PDF dated 5/9/2019, it shows bug PAN-105737 as fixed, but just downloaded the PDF again and it's dated 5/23/2019 and that bug is back under "Known Issues".

Did you try the workaround of adding a default gateway to the AUX interfaces in your case?

[BPry]

@Pzilla,

The workaround worked on one of my HA pair 5220s, but failed on two different HA pairs. I'm not sure how much faith I would put on the gateway actually being a fix for the issue.

That was actually why I installed 8.1.8, as it was supposed to be fixed.

[Pzilla]

@BPry ,

Thanks for confirming the workaround didn't work for two of your three 5220 pairs.

We only have one 5220 pair but it's production so looks like I'll be waiting for 8.1.9, and hopefully the bug is actually fixed this time. 

[Pzilla]

Looks like 8.1.9 was released yesterday, and the release notes now list the bug as fixed ("Addressed Issues"):

PAN-105737 (PAN-OS 8.1.7 & 8.1.8 only) Fixed an issue where AUX ports remained in Down state after you upgraded to PAN-OS 8.1.7.

Anyone with a PA-5200 series HA pair in a non-prod environment had a chance to test yet?

[arun_sh]

Okay. so even We were hit by this bug. we have two 5220 PA FW in HA and HA1 connected via AUX. We were in plan to upgrade from 8.0.10 to 8.1.7 but then while secondary was upgraded and rebooted to 8.1.7 , split brain scenario was occured and we had both our firewall in active active and whole network was down. 

   Then we get to know from PA that its a bug. then we proceeded for 8.1.9 upgrade and it was resolved.

  but now I have a question and looking for solid answer here- this bug only caused HA1 link to remain down but then what about the HA1 backup hearbeat, as we had heartbeat backup enabled in election settings and by default it takes mgmt port. Hence if HA1 was down due to bug, why backup heartbeat did not function through mgmt and made sure that HA can be acheived. could you pleaee help?

[Pzilla]

@arun_sh , thanks for confirming that 8.1.9 works for HA on your 5220 pair.

I'm thinking of scheduling an update to 8.1.9 now that bug PAN-105737 has actually been fixed, along with the fixes for two of the three security advisories published yesterday.

I'm not sure why the management port was not working for HA1 backup, the bug description only mentions AUX ports being in a link down state. I'd think that should be a question that PAN TAC would be able to answer.

On our 5220 pair we are using AUX1 and AUX2 for HA1 and HA1 Backup as I like to keep management and HA functions separate if the hardware has the capability, but 5220's also have RJ45 dedicated HA1-A and HA1-B 1Gbps ports as well which you could look into using.

[arun_sh]

the reason why some of the Palo Alto firewalls in HA are facing this split brain scenario is due to this BUG -106914.

this is mentioned in 8.1.9 PAN OS as addressed issue.

  please find the detail:

Fixed an issue on a firewall in a high availability (HA) active/passive configuration where HA1 and HA2 links stopped passing packets, which caused a split-brain condition after an automatic configuration sync.

hence , either you are using AUX or mgmt port or any other port, if this bug is hit then HA will stop working and FWs will become active active. I am sure my config though having AUX for HA1 but mgmt for HA1 backup(not physical, but through election settings), was hit with this BUG as well.

  please do upgrade HA FWs with downtime only as there is no mention of any particular hardware model.

[Pzilla]

I updated our 5220 pair to 8.1.9 during an evening maintenance window a few days ago and no issues with split-brain so far.

[BPry]

@Pzilla,

I've been running it on production equipment since last Friday without issue, and testing it in a lab environment since slightly before public release. It does appear that 8.1.9 fully addresses this issue as promised. 

[Srikanth.Singireddy]

Sorry everyone for digging up an old grave.

I have exactly the same issue and I am running 9.0.8. I tried upgrade and downgrade and still no fix.

I have no clue what went wrong with my config , but the backup ports when using aux2 is always down.

TAC seems clueless as well.. Any suggestions?