Failed active/passive HA Upgrade from 8.1.4-h2 to 9.0.3

[SteveBallantyne]

Hello community - I have a case open with support, but I am looking to see if anyone else has an idea for me while they are looking at my tech support files.

I attempted to upgrade an active/passive HA pair following the Palo Alto Doc. I upgraded the secondary from 8.1.4-h2 to 9.0.3 and rebooted. I am now at the step where I would suspend the primary and fail over to the secondary. But my HA pair is broken.

Here is what shows on the Dashboard of my primary ...

Mode		Active-passive
Local		Active
Peer (172.17.1.11)		Unknown
Running Config		Synchronized
App Version		Unknown
Threat Version		Unknown
Antivirus Version		Unknown
PAN-OS Version		Match
GlobalProtect Version		Unknown
HA1		Down
HA1 Backup		Down
HA2		Down

I find it odd that it shows a version match for the PAN-OS Version, when in fact, the secondary is running 9.0.3?

[SteveBallantyne]

Also - my secondary now shows "HA not enabled" on the Dashboard, even though it's still configured?

[SteveBallantyne]

I ended up upgrading (or in this case *downgrading*) my secondary to 9.0.0 and that worked. So then I was able to upgrade my primary to 9.0.0 without any issues. None of my VPN tunnels came back up on their own, which was a little disheartening. I had to go into the CLI and do a "test vpn ike-sa gateway <tunnel name>" on every single one of them.

But my upgrade is complete and functional for now.

[BPry]

@SteveBallantyne,

Just out of curiosity, what did you upgrade path actually look like? 

In this situation you should have followed the following path to meet best practices:

8.1.4-h2 -> 8.1.9 (As the latest maintenance relase) You do not need to restart (I would anyways)

8.1.9 -> 9.0.0 Install and Reboot

9.0.0 -> Target Maintenance Release (9.0.3) Install and reboot 

[blawrimore1]

i too have this condition while upgrading from 8.1.9 to 9.0.3-h3.  the release notes and upgrade guide state i can upgrade directly to 9.0.3-h3 without the intermediate 9.0 step.  however, this post led me to downgrade to 9.0 from 9.0.3-h3 and re-attempt a non-impactful upgrade.  this did not fix my situation but i was able to continue my upgrade path, just with impact to user traffic.

upgrade to 9.0 was impactful, but after both devices upgraded, HA2 came online and synchronization was successful.  upgrade to 9.0.3-h3 from 9.0 was hitless and uneventful.

also, i needed to upgrade my logging server to 9.x before logs would start showing up in panorama for this set of firewalls.

[David-P]

Bug 128629 is what we ran into when upgrading from 8.1.6 to 9.0.5

Only the HA2 HSCI link was down for us.

setup a HA2 Backup link.

then was able to proceed without interruption.

PAN-128269

PA-5250, PA-5260, and PA-5280 firewalls with 100GB AOC cables only

Fixed an issue where after you upgraded the first peer in a high availability (HA) configuration to a PAN-OS 9.0 release, the High Speed Chassis Interconnect (HSCI) port did not come up due to an FEC mismatch until after you finished upgrading the second peer.

[jeremy.larsen]

If you read the upgrade guidance "carefully", you will see it is recommended to upgrade to the latest code train release before jumping versions.

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-new-features/upgrade-to-pan-os-90/upgrade-the-fi...

"Download and install the latest preferred 8.1.x maintenance release and reboot."

8.1.4 -> 9.0.3 is a HUGE jump that I would never attempt.  I would have gone to at least 8.1.11 or whatever was available first, then you can leap over 9.0 (as long as it's downloaded) and go straight to 9.0.3 (current recommended it 9.0.5).