Panorama PAN-OS Upgrade from 9.1.10 to 10.0.7 stuck at 45%

[Roberto_F]

I'm trying to upgrade a Panorama M-600 appliance from 9.1.7 to 9.1.10 (this step succeeded), to 10.0.7. But for the past 2 hours the upgrade has been stuck at 45%:

admin@iReport> show jobs id 7330
Enqueued              Dequeued           ID                              Type                         Status Result Completed
------------------------------------------------------------------------------------------------------------------------------
2021/09/10 22:13:24   22:13:24         7330                         SWInstall                            ACT   PEND        45%
Warnings:
Details:

I've been on hold for over an hour with Palo, so will try to post the question here. Should I kill the job or reboot? Or do I need to wait as there's TONS of data on the appliance?

FYI while Panorama was stuck I upgraded our two PA-5220's from 9.1.7 to 9.1.10 to 10.0.7 without any issues. I know I had to do Panorama first... but I was optimistic about its upgrade succeeding as well.

Thanks,

Roberto

[Roberto_F]

Wow... I'm sooooo glad I didn't jump the gun on rebooting. The upgrade just completed - it took 2hrs and 20 minutes. Life is good again.

[BPry · Accepted solution]

@Roberto_F,

Just for the future, seeing an M-600 take an extended amount of time to perform major version updates is expected behavior. During the upgrade from 9.1 to 10.0 every single log file on the M-600 has to be migrated, so the more logs a system has the longer the process is going to take. I've seen some updates take upwards of 5 hours. Maintenance updates are much quicker because you don't have to do anything to migrate the logs, as they are all already in the proper format. 

Also I'd also put out that I personally wouldn't recommend upgrading your M-600 (presumably functioning in Panorama mode) and your firewalls at the same time. If you run into a bug or issue post upgrade and need to downgrade your M-600 you would also need to downgrade your PA-5220s at the same time so they can be managed by Panorama. 

I would recommend upgrading Panorama the week prior to upgrading any of your firewalls. This allows you enough time to see if you'll run into any issues on the Panorama upgrade prior to upgrading any of the firewalls, so you don't have the risk of needing to downgrade multiple devices at the same time. 

[Roberto_F]

In addition to BPry's comments (completely accurate), to help anyone running in the same issues after the looong M-600 Panorama upgrade, the M-600 did not show any more firewall logs after the reboot. Neither real-time nor pre-existing logs were being reported. After an hour-long remote support session with Palo's support, they too were at a loss as to the root cause. They collected Panorama's logs to review them offline.

The morning after (about 7 hrs later) I checked again just in case... and Panorama was now showing the real-time traffic and had a couple of months worth of past logs. It took the rest of the day, but by the evening all historical logs had reappeared as well.

So don't panic if you don't have data after the v9 - v10 upgrade for several hours. They will eventually appear.

[AmolMohite]

I have also faced same issue... It is usual behavior of upgradation to major software updates... I have done 9.1.4 to 10.1.1 upgrade . its stuck at 45% for more than 20 min for each upgrade.

[AmolMohite]

I have also faced same issue... It is usual behavior of upgradation to major software updates... I have done 9.1.4 to 10.1.1 upgrade . its stuck at 45% for more than 20 min for each upgrade. My OS upgrade is done successfully.

[grenzi]

Hi everyone and thanks a lot for the useful information reported in this thread. I hope I'm not off topic, but does anyone know if the same issue (long upgrade time due to log migration) applies also to the VM version when it's in legacy mode?