topic Re: Upgrading 4.07 to 4.1.2 in HA environment in General Topics

Upgrading 4.07 to 4.1.2 in HA environment

gmoerschel — Thu, 26 Jan 2012 17:09:20 GMT

The following change log may be useful to all of you wondering how an upgrade goes in an HA active-passive pair. It would be nice if PAN support were to put this into a tech note. Each step is essentially a check or an observation from top to bottom.

2050 Firewall Upgrade 4.07 to 4.1.2 Log:

Pre download of PAN-OS 4.1.0 and 4.1.2 to both units

No commits pending

firewalls shows HA is in synch

Same version of dynamic updates are installed on both units

Backup configs from both and export them

On PAN2 (the current passive), suspend the unit via GUI:device/ha/suspend @ 9:17 am

PAN2 upgrade to 4.1.0 starting @ 9:18am

PAN2 reboot @ 9:22

PAN2 back up at 9:25 and auto com until 9:38. PAN2 comes up as non-functional state. Note that the unit will not be able to log you in for several minutes as the upgrade process is happening.

PAN2 Started 4.1.0 to 4.1.2 upgrade at 9:40

PAN2 reboot at 9:44

PAN2 back up at 9:50 and auto com until 9:53. PAN2 comes up as non-functional state.

Because 4.1.x HA is not compatible with 4.0.x HA there is no way to make the newly upgrade firewall the active. The older fw code must be suspended first and then the new fw is made functional which then brings it into the active state. Because of this limitation, the sessions going through the active will sever when it is suspended. Detail below:

do quickly PAN1 from cli: request high-availability state suspend

NOTE: active sessions will sever when active becomes suspended

do quickly PAN2 from cli: request high-availability state functional

PAN2 immediately becomes active. loss of 14 pings

PAN1 upgrade to 4.1.0 starting @ 10:56am

PAN2 (not PAN1!!) redefined pre-emption to 98 to be favored over PAN1 so that PAN1 does not become active upon reboot into initial 4.1.0 install.

PAN1 reboot @ 11:04

PAN1 back up at 11:10 and auto com until 11:20

PAN1 state shows as "initial" after auto com completes

PAN1 Started 4.1.0 to 4.1.2 upgrade at 11:23

PAN1 state transitioned to passive while 4.1.2 upgrade was running

PAN1 reboot at 11:27

PAN1 back up at 11:32 and auto com until 11:35. PAN1 comes up as passive.

HA dashboard widgets on both sides show no errors and states are synched.

PAN2 pre-emption number set back to 101 then PAN2 was suspended. At this point an auto com job started running on PAN2 after PAN1 became active. This job took about 1 minute. This may be a synch check.

PAN2 commit config (to set back preemption value to 101)

PAN2 "request high-availability state functional" and state then becomes passive

Process completed at 11:43 am

Post upgrade items to do:

Download GlobalProtect client

Activate GlobalProtect client

Enable user ID on outside zone per error message (probably a Global Protect requirement)

Ensure User-ID agent reconnects

Test GlobalProtect upgrade and function

Re: Upgrading 4.07 to 4.1.2 in HA environment

jdelio — Mon, 06 Feb 2012 17:27:00 GMT

Thanks for posting this and adding to the discussion here.

Kind Regards

Re: Upgrading 4.07 to 4.1.2 in HA environment

User_333 — Wed, 08 Feb 2012 08:15:04 GMT

Isn't it possible to upgrade directly to 4.1.2? Why is this intermediate step to 4.1.0 needed?

Re: Upgrading 4.07 to 4.1.2 in HA environment

gafrol — Wed, 08 Feb 2012 09:56:20 GMT

Unfortunately you cannot upgrade from 4.0.7 to 4.1.2. You have to upgrade to 4.1.0 first otherwise the upgrade will fail...

rgds Roland

Omicron AG - Wallisellen 🙂

Re: Upgrading 4.07 to 4.1.2 in HA environment

MichaelWright — Wed, 22 Feb 2012 05:25:14 GMT

This is very well put together. Thank you for taking the time to do this for everyone.

Re: Upgrading 4.07 to 4.1.2 in HA environment

gmoerschel — Wed, 22 Feb 2012 14:16:54 GMT

See notes in original thread that you must go to 4.1.0 first which is the basis for that code train. 4.1.2 includes modifications to the base code.

Re: Upgrading 4.07 to 4.1.2 in HA environment

migration — Wed, 22 Feb 2012 19:09:46 GMT

Correct me if I am wrong but you only have to download the 4.1.0 code? I upgraded a couple of 5020's that way last week.

Re: Upgrading 4.07 to 4.1.2 in HA environment

User_333 — Thu, 23 Feb 2012 21:00:47 GMT

Yes, it's definitely not needed to INSTALL 4.1.0 in front of a upgrade to 4.1.2. The only requirement is a downloaded 4.1.0 Firmware. This is described in the Admin Guide (Page 38) too:

You must have a base image downloaded before you can install an update version. For example, you must have 4.1.0 downloaded (not installed) before you can upgrade your 3.1.9 device to 4.1.4.

Re: Upgrading 4.07 to 4.1.2 in HA environment

licenselu — Fri, 24 Feb 2012 07:50:45 GMT

Hello,

In the initial post, I saw that the timeframe between the reboot and the autocommit is 15 minutes !!

It's the normal boot time for PA 2000 series ??

PS : all my PA 500 series boot in less than 6-8 minutes. Twice for PA 2000 series ??

Regards,

Hedi

Re: Upgrading 4.07 to 4.1.2 in HA environment

DanDre — Tue, 29 May 2012 05:18:14 GMT

Thanks for the information! Just did this on our 5050-clusterpair without any problems.

Re: Upgrading 4.07 to 4.1.2 in HA environment

tgeorge — Thu, 05 Jul 2012 21:25:29 GMT

Thanks, followed your document to perform the upgrades on our PA-500 HA Pair. Much appreciated.

Re: Upgrading 4.07 to 4.1.2 in HA environment

BSadozai — Thu, 17 Jan 2013 15:50:12 GMT

Hello everyone,

I wanted to upgrade our PAN 2050 yesterday evening from 4.1.8 to 5.0.1
directly and had issue with error message on my active unit seeing that (peer
non functional and peer version no compatible or not the same as local,
something like that).

After searching and didn't find any doc that explain the process upgrade
from 4.1.x to 5.0.X I did a downgrade, and today I saw this discussion that is
very helpful. do you thing that I have follow the same process described here
to upgrade to 5.0.1 and do you thing that is really mandatory to pass first by
5.0.0 before going to 5.0.1.?

Thanks for your help

BES

Re: Upgrading 4.07 to 4.1.2 in HA environment

mbutt — Thu, 17 Jan 2013 16:09:41 GMT

Hi BES,

When upgrading to a major release in you case from 4.1.8 to 5.0.1 you have to always install the base release of the Major release.

So in other words when upgrading from 4.1.8 you first have to download 5.0.0 and then download and install 5.0.1.

Hopefully this helps.

Thank you

Numan

Re: Upgrading 4.07 to 4.1.2 in HA environment

BSadozai — Thu, 17 Jan 2013 16:17:12 GMT

Thanks for your quick answer, but in fact I already did that, i mean I downloaded on both my devices active and passive first v5.0.0 (without installing as was specified by the device, just download) and after that I downloaded the v5.0.1 and installed on passive unit, but after the reboot of the passive unit I had the status non fonctional and on the active I saw the HA status ( something like peer non fonctional (peer version nocompatible or mismatch) so I did the downgrade.

Could be very helpfull from Palo to have a official process or doc for that, I really dont understand way there is no doc for that.

thanks alot

BES

Re: Upgrading 4.07 to 4.1.2 in HA environment

msullivan — Thu, 17 Jan 2013 21:40:19 GMT

Here is a nice 'how to' doc:

https://live.paloaltonetworks.com/docs/DOC-4043

Cheers,

Mike

Re: Upgrading 4.07 to 4.1.2 in HA environment

BSadozai — Fri, 18 Jan 2013 09:29:44 GMT

Hello,

Thanks for the link, but I followed the process described on
this doc till point 4. And I had the status of the HA (no fonctional “peer
version mismatch or not the same”) and
at this point my understanding is that the HA is not functional and if I went
to point 6 to suspend the second device I
have had a loss of connectivity and didn’t know if the upgraded device will
take the active role and if yes after how many time, as I didn’t plan on this
evening any downtime of our internet line I did a rollback

First suspend the active unit
from the CLI run the command:
> request high-availability state suspend

or

From the GUI go to Device > High Availability > Operations > Suspend
local device.

Note: This will cause an HA failover. It is recommended to do this first
to verify the HA functionality is working before initiating the upgrade.
Verify network stability on
the new active device with the previously active device suspended.
Install the new PAN-OS on the
suspended device, then reboot the device to complete the install.
When the upgraded
device is rebooted, the CLI prompt should show passive
(or non-operational, if on a different major release ie 4.0 to 4.1) and the
PAN-OS version should reflect the new version.
On current passive device,
verify auto commit completes successfully (FIN OK) by running command: show
jobs all before proceeding to the next step.
Suspend second device (should
be current active device).
Upgrade the second device,
then reboot it. When second device restarts, the first device that was already
upgraded takes over as active.
As HA functionality was
verified (step 1) and the config was successfully pushed to the dataplane on
the new PAN-OS (step 5), the failover should be seamless.
When the second unit reboots
it will come up as the passive unit. Validate the auto commit completes on this
device by running command: show jobs all
on this device (as done in step 5) to complete the upgrade. The original active
device before the upgrade will be the active device now.

Thanks for your comment

BES

Re: Upgrading 4.07 to 4.1.2 in HA environment

msullivan — Fri, 18 Jan 2013 17:12:21 GMT

I get why you'd be concerned, and the document could be improved by describing what is expected to happen at step 6. As you stated, it appears like you have a broken cluster and so FUD creeps in and so you are reluctant to pull the trigger and finish the upgrade.

In reality, all is well and when you suspend the active device at step 6, the newly upgraded device will take over, it just isn't documented or explained anywhere that I've seen. You might drop a couple of pings but that's it.

I'd like to hear from PAN support to explain what's going on at step 6. If I had to guess, it would have something to do with the HA interfaces being smart and taking action on the cluster, even though it looks broken at the time.

Open a support case and talk to them about it, then give it a shot!

Cheers,

Mike