auto commit issues after upgrade to 10.x

[RREALICA]

Hi,

We started to experience auto commit finishing delay on our PA-5220 after the upgrade to 10.x.  We have a pair of HA PA-5220 in active/passive mode, we never had an auto commit issue before in previous updates, reboots of the firewalls. We have upgraded numerous times before from 8.x all the way to 10.x.  In our recent upgrade to 10.1.x, three of the four firewalls failed on the initial auto commit, of those two of the three eventually finished after retrying a few times in about 10 minutes, one of them though took about 60 minutes to complete.  

I know it's published that it may take 30 minutes for auto commit to finish, but in our case we actually never seen it go over more than 5 minutes in the PA-5220 until the upgrade to 10.x.  When it was failing to auto commit the following error was present on all three firewalls but did eventually cleared by itself. i

configured traffic quota of 0 MB is less than the minimum 32 MB.
Invalid configuration. Please fix errors and try again.
Failed to commit policy to device

in our case this was not service impacting, since it was on HA pair but we do have standalone firewalls that if they are stuck at auto committing then it would be service impacting.

Support basically said this is acceptable/normal.

I just want to know if this is the new normal now for us and to set expectations as such, and what others experience is with auto commit finishing. 

Thanks.

[kiwi]

Hi @RREALICA ,

This could be expected after 10.1.x due to some changes, autocommit fails unless all of the logging disks are ready and have stopped rebuilding.  Expected times for rebuilding the disk will depend on the logging size itself (I've seen it take 60 - 90 minutes).  Allow time for the rebuilding to complete.

Hope this helps,

-Kiwi.

[fwmike2]

Thanks @kiwi.
Experienced a 30 minute reboot to HA ready on PA-5220 going from 10.0.11-h1 to 10.1.6-h6.

Saw the same auto commit errors as @RREALICA.

It did not seem possible to SSH into CLI as admin (to check RAID status) until the system completed the auto commit. 

[codyh]

We also ran into the same issue on an HA Active/Passive pair of PA-5220s. Our upgrade path was 9.1.14-h4 -> 10.0.11-h1 -> 10.1.8. We noticed on the dashboard that HA was down, and also all of the interfaces showed down and not configured. In Tasks we were getting the Auto Commit failure repeatedly with details of:

configured traffic quota of 0 MB is less than the minimum 32 MB.
Invalid configuration. Please fix errors and try again.

Our uptime was 87 minutes when the Auto Commit finally completed. Once that was done everything appeared to be working as expected. Only one firewall in the pair had this happen, the other upgraded in a normal time window (15ish minutes).

As a note, you can check on RAID with cli command: show system raid detail

[fwmike2]

A pair of PA-5250 10.0.11-h1 -> 10.1.6-h6 took ~67 mins per device.

[jkim12]

5410 from 10.2 to 10.2.3 upgrade and enabling HA took 2hr:01 min 

[KTompkins]

Hi Kiwi, could you share what CLI command you ran on the firewall to get the output you show in your screenshot? Thanks in advance.

[kiwi]

Hi @KTompkins ,

The CLI command is :

> show system raid detail

Here's the full KB article: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000wkxPCAQ

Kind regards,

-Kiwi.

[SimonCoggins]

Thanks for including your times. I have 4 x 5220s (2 HA pairs) and managed to hit this on all 4. Your timings assisted in me greatly. Mine were 82mins.

This was going from 9.1.11 -> 10.0.11 -> 10.1.10

[Mohsinh]

I upgrade 5220

It took 45 Mins around for Auto Commit to get completed from failed state till that interface was down. 

Raid System disk was in progress even after interface came back and took 8 hours to get 100% completed

We upgrade from 9.1.6 >> 9.1.11 >> 10.0.11 >> 10.1.10-h1

[stevess]

I'm upgrading an HA pair of 5250's tonight and ran into the RAID rebuild issue on the primary firewall but not the secondary. This is the second time that has been the case, so my theory is that the primary has a much more full disk than the passive firewall (since it rarely runs traffic to fill the logs) so perhaps that is why the passive firewall either finishes too fast to notice or doesn't do the rebuild at all. I'm currently at 98 minutes with no auto-commit completion yet. Its going to be a long night.

[ronsmith35]

We upgraded a pair of 5220s from 10.x to 10.1.4 and the primary took over 4 hours.  I'd recommend clearing your logs before you do an upgrade so the raid rebuilds take less time.  The remaining passive HA 5220 took 15 mins.  I think this happens also when you go to 10.2 from 10.1.x...Can't wait until I get a maintenance window for that....eck.

[Anders_Bohman]

We have seen same behaviour happen to some of our PA-52xx hardware during 10.1 upgrade, not all of them. I have opened a ticket to find out if there is any way to foresee this raid rebuild in advance, or take some other actions before the upgrade to prevent it from happening. If anyone know a way, i would be happy to know 🙂

[JimSilha]

I hope to add some color here based on our experience with this issue and discussions with TAC and PAN dev team.

Upgrade 9.1x to 10.0.x

There is a RAID rebuild that takes place on the 2 TB log disks. PAN reports that this is due to a file system change that happens during the upgrade. The following KB article sort of addresses this but indicates this happens in 10.1, however, in our experience and TAC reports that this occurs in 10.0. During our upgrade of an HA pair of PA-5250's, the auto commit completed fairly quickly but the RAID rebuild took 12 hours to complete. We were cautioned not to reboot the firewalls during the RAID rebuild and upgrade to 10.1.x once the rebuild was complete.

auto-commit error: "Configured traffic quota of 0 MB is less than the minimum 32 MB" after upgrade to 10.1.4 or later.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000wkxPCAQ

Upgrade 10.0.x to 10.1.x

There is a an FSCK check (disk integrity check) that happens every 8th reboot or if the firewall hasn't been rebooted in more than 90 days. PAN TAC can view the status of this via root access. This is also mentioned in the Upgrade/Downgrade Considerations for 10.1, linked below. TAC states this could take 60 - 90 minutes to complete before the auto commit will complete. In our experience, this can take far longer, 5+ hours. The duration is dependant on the ammount of data (logs) on the disk.

See: PA-5200 Series, PA-7000 Series, WF-500, and WF-500-B Firewalls

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-upgrade/upgrade-pan-os/upgradedowngrade-conside...

[MP18]

@JimSilha 

We had same issue now. PA 5220 in HA auto commit is not working due to same error as mentioned above.

Already more than 2 hours and waiting not sure how long it will take.

Already had a long day and still need to upgrade HA firewalls to 10.2.4h4.

Regards