Unable Sync Configuration between HA Pair after downgrade from PANOS 10 to 9.1.7

[cpartsenidis]

Hello everyone.

I'm stuck in a bit of an odd situation here with my two PA-850 firewalls unable to sync between each other. 

Initially both firewalls were on PANOS 9.1.7.  Fully syched and configuration backed up.

The problem occurred because I upgraded the secondary-standby unit all the way to 10.1.6-H6 and then had to rollback to 9.1.7.The primary-active unit was not touched during this process.   After having the secondary-standby unit rolled back to 9.1.7, I noticed that I was not able to sync the running config between the two firewalls.

Palo TAC unfortunately haven't been very useful and despite providing tech-support files of both firewalls, the assigned engineer failed to notice the problem.  Researching further into the issue, I came across the 'show high-availability all' command which help reveal the problem:

Configuration Synchronization:
Enabled: yes
Running Configuration: not synchronized
Out-of-sync Reason: Version mismatch with Peer for DLP

After sharing this with Palo TAC, they suggested to upgrade both firewalls to 9.1.8 as the Auto deletion of DLP directory/plugin on downgrade to 9.1 is only fixed in 9.1.8.  So I upgraded the Secondary-standby firewall unit to 9.1.8 and restored the original 9.1.7 configuration to it, however the problem still persists.

I don't want to upgrade the Primary-Active unit at all, until this situation is resolved.

Can anyone provide some guidance or assistance?

Many thanks,

Please note you are posting a public message where community members and experts can provide assistance. Sharing private information such as serial numbers or company information is not recommended.

[cpartsenidis]

Just thought I'd reply and post back the solution I tried and worked for me.

To cut a long story short, I upgraded the secondary-standby firewall to 10.0.0, manually removed the DLP plugin using the CLI command "request plugins uninstall dlp", then downgraded to 9.1.7 and it worked as expected.  I could sync configuration between the two firewalls again.

TAC support was not helpful as I ended up finding the root cause and took a different approach to the one they provided (which was to upgrade both firewalls to 9.1.8 - which wouldnt have worked anyway).

Thanks.

[aleksandar.astardzhiev]

Hi @cpartsenidis ,

Thanks for sharing your experience. Can you share the reason why you rollback from 10.1 back to 9.1?

[cpartsenidis]

Apologies - I should have made it clear why I rolled back.  I upgraded the secondary standby unit from 9.1.7 all the way to 10.1.6-H6 but then couldn't fail over because of the major differences in the PANOS versions, so I wanted to rollback the change and start again, but hit the issue of the two firewalls unable to sync their config even after the secondary standby unit was back on 9.1.7.

Reason they were unable to sync was because of the DLP plugin which got installed from PANOS 10.0 onwards 😉

Hope that makes it clear now.

Cheers,

[BPry]

@cpartsenidis,

Just going forward; I know a lot of people don't like the maintenance window required to properly upgrade multiple major versions in an HA pair, but you really shouldn't allow the peer members to be more than one major version ahead of each other. 

There's plenty of people that do exactly as you've done and then just bring the primary firewall on the older code down manually and remove it to get it back in sync with the updated secondary firewall. There's additional risk when you follow this process that I'm not personally a fan of if you have a choice of just extending the maintenance window to walk them both through the required upgrades. 

[rsivasanka]

When firewall was upgraded to 10.0.x  and then rolled back for some reason.

• Upgrading the firewall to 10.0.0 Enterprise DLP automatically gets installed. 
• You must uninstall Enterprise DLP before you can successfully downgrade from PAN-OS 10.0 to an earlier release. 

admin> request plugins uninstall dlp

https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-new-features/upgrade-pan-os/upgradedowngrade-co...

[jasonroy]

Some of us have standardized deployment versions and don't chase the new shiny.  But when devices fail and have to be replaced there can be issues such as when the RMA unit arrives and has 10.1, there is a need downgrade to 9.1 (10.1.x -> 10.0.0 -> 9.1.0 of course).  First, we need to issue this command, otherwise once downgraded to 9.1 the command isn't available, but yet the two HA units can see the DLP doesn't match (one has no DLP version, RMA unit has DLP version).

The request plugins uninstall dlp command when running 10.x.x before downgrading to 9.1 does the trick.  This is noted in the referenced document:

https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-new-features/upgrade-pan-os/upgradedowngrade-co... 

[RBozdemir]

Hi,

If you remove DLP on both sides, you will see that sync is working.

[C.Shirkoff]

Your experience with TAC is similar to mine. Whether your issue is solved by TAC is hit and miss, as it seems there is a wide range of skills in the tier 1 engineers. I have had multiple issues with Global Protect LSVPN which apparently is a feature few TAC engineers have experience with, and often find the solution while TAC is requesting yet another techsupport file or tcpdump.

Thanks for sharing your solution and the reason for your rollback as this information is invaluable to the community.

[HighMiles]

Thanks Cpartsenidis for sharing the solution to this issue, that is exactly what I've encountered when upgrading from 9.1.11.h4 to 10.2.8, it broke HA and had to downgrade back to 9.1.11.h4 and got the "version mismatch with Peer for DLP" knowing that i am not running DLP!

Thanks again