Auto Commit stuck at 11.0.2-h2 PA-410

[Ankit1Singh]

We have new PA-410 Device which comes with pre-install PAN-11.0.2-h2.

FW Auto commit keeps failing and starting again & again. 

Steps we tried: ---

1. Firewall reboot.

2. Factory reset done.

3. Tried to downgrade the PAN-OS to previous version but it failed with error " Auto-commit in queue and cannot process this task".

Please suggest if any solution to mitigate this issue.

[pxsecurity]

I created a case for a similar issue at Palo Alto with my PA-410 FW. 

According to the support engineer, who confined the bug with development, this issue will be fixed in 11.0.3 (ETA is 11/02/23)

[Bharath_A]

Hi Pxsecurity,

Could you please share the PA case number if possible for the reference. That will be helpful for me.

[Giovanni_Russo]

I'm experiencing the same problem with the PA-410 firewalls, but upgrading to version 10.1.11-h1, I've tried both rollback and a factory reset, but the autocommit always stays stuck. Can you give me a solution?

[Ankit1Singh · Accepted solution]

I have worked with TAC on this issue and we performed below steps to fix this issue. Hope this helps.

           >> Entered into maintenance mode, reverted PAN-OS to the 11.0.0 base version and did a factory reset.

      >> Auto commit was successful, and we are now able to configure the management IP.

      >> No auto-commit issue on Pan-OS 11.0.2-h1. Kindly upgrade to it.

      >> As per Engineering team, the issue is fixed in PanOS versions: 11.0.3, 11.0.4 and so on...

[Bharath_A]

Hi All,

We requested customer to Downgrade to 11.0.2.-h1 and then commit was successful. Till now I didn't see any reported bugs in Palo Alto portals.

Thanks

[MCervenka]

We had the same problem with 10.1.11-h1 on 410 devices. Unfortunately on five devices at once and all of them were IPSec remote sites. 
What helped was reboot into maintenance and revert to the previous version (there were different 10.1.X). 100% upgrade failed on 410 devices.

[Will-78]

I have experienced this with a PA-410 going to 10.1.11-h1 as well. I checked known issues and resolved (in 10.1.11-h3 and h4 addressed issues) and did not find any reference to this bug.

FYI, my disk images and factory reset did not show any revert options. I had to go into Advanced Options (password MA1NT) to finally see revert-able images to downgrade out of the bug.

[jasonroy]

Had this problem on the way from going from 10.2.x to 11.1.x. The upgrade from 10.2.x to 11.0.0 was fine. Upgrade after 11.0.2-h2 had the problem as mentioned: stuck in auto-commit loop. I used this article for help reverting:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PNnJCAW 

Using the debug swm status that 11.0.0 was listed as REVERTABLE:

> debug swm status

Partition State Version
--------------------------------------------------------------------------------
sysroot0 REVERTABLE 11.0.0
sysroot1 RUNNING-ACTIVE 11.0.2-h2
maint    READY 11.0.2-h2

Using the debug swm revert command caused the 11.0.0 image to be installed. A job type of "SWRevert" showed "FIN OK" and debug swm status showed it was now ready to go:

> debug swm status

Partition State Version
--------------------------------------------------------------------------------
sysroot0 PENDING-REVERT 11.0.0
sysroot1 RUNNING-ACTIVE 11.0.2-h2
maint READY 11.0.0

After a request system restart it successfully booted back down to 11.0.0.  At this point I was able to upgrade to 11.0.3 and the auto commit had no problem. Once upgraded to 11.0.3 I was able to upgrade to 11.1.0 as expected and on to the latest 11.1.x.

[jasonroy]

Side note: kicking myself as the 11.1 docs show 10.2.x is acceptable to upgrade directly to 11.1. There was no requirement to go to 11.0.

https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-upgrade/upgrade-pan-os/upgrade-the-firewall-pan... 

[J.Patel]

Hello All,

Well this is interesting, I have a PA-415.  since around March 2023. The unit was shipped with 11.0 I have never had to upgrade anything on the unit as the unit was updated. The firewall has been running for 1 year and no problems so far.

Last week, I performed a software check on a whole list of firmware revisions was shown, I was hoping for 11.1 to be shown. I downloaded the firmware an went straight to upload. Unfortunately this failed.

I then went down a firmware to "h" models and about 5 previous firmwares failed.

BTW At this point, the download and update on each firmware lasted between 15 to 20mins.

I must have gone down to one of the first revisions of 11.0 which in fact I worked, I even checked on the support portal for firmwares which was not listed on the firewall and didnt work. So overall we are talking about 7 variations of firmware between the support poral and onboard firmware.

What i noticed after downgrading the firmware, a message was shown that I had to update the databases on the security services, the question was which one? The AV, Threat protection, wildfire, URL, ??? Nothing was mentioned again. So out of guess work, I had to manually download the signature database manually of each security services from the live community I found on several threads. Again this tool me about 15mins to 20mins on each file.

By this time im looking a 1hr gone.

After several attempts, I mange to upgrade the firmware from 11.0 to 11.0.3 with a new database, then I had to upgrade to the 11.0.3h version, then upgrade to the 11.1 version.

So overall all your looking at 2hrs. This was all last week, week commencing 29th January.

This week from 5th February, I was researching more into the ML In-Line services, what I found curious, that all the IN-Line were not active. You have to create a new entry into each of the CDSS services and enable the IN-Line services for each one. Then the recommended path is to change the default state to reset both client and server.

To me, if the In-Line services is the key function, service, technological key component of the firmware, shouldn't it be on default so everything is updated in real time and stops the user for making several of configuration changes on the firewall? As a business, having realtime protectin is a simple yes and attribute to any business.

Then if the firmware has to be updated to 11.1, at least have instructions on how to update the database of the services to get to 11.1 or have the 11.1 pull down an updated database also for ease of installation.

Wanted you to know that the upgrade path for me was tricky but its all working now as expectated.

Thanks

Jatin

[L_Carrera]

Can confirm that the bug is still on 11.0.2-H3.

Newer devices managed by Panorama (11.0.2-H3), locally upgraded through portal.

From 11.0.0 to 11.0.2-H3 on single FWs was fine. Three so far.

But on my first HA cluster, the secondary passive FW went into the Auto Commit loop after reboot.

Rebooted and tried canceling job ID from portal, of no help.

From CLI after a few attempts, was successful able to start the reinstall of 11.0.2-H3

show jobs all

clear job id <???>

then as fast as possible

request system software install version <11.0.2-h3>

once completed manually reboot the device.

Hope this helps.

[derrickmertz]

Has anyone seen this on 10.1.11-h5? I have three PA-410s that are now stuck in this state after upgrades from 10.1.9-h3. Has anyone tried upgrading to 10.1.12 instead of going to 11.0, and was that successful?

Thanks

[Will-78]

10.2.7-h3 and 10.2.8 will definitely work and are not 11.0 (with so few maintenance releases). 10.1.12 is supposed to fix it but I haven't tried it.

As a warning, 10.2.7-h3 and 10.2.8 broke my log forwarding to Panorama. I had to add the firewall to a log collector preference list (even though I only have the one Panorama in Panorama mode) *AND* use a custom cert to get it to work again. At the time I figured this out PAN had a few cases open with my similar symptoms and no resolutions yet.

[Will-78]

One other thing to think about, PAN fixed 10.2, 11.0, and 11.1 pretty quick but the official fix for this in the 10.1 train was notably later. This made me think that they are de-prioritizing the 10.1 train now. It's possible I'm wrong but that thought process encouraged me to get to 10.2 on all of my firewalls, despite not needing the new features.