auto commit issues after upgrade to 10.x

[RREALICA]

Hi,

We started to experience auto commit finishing delay on our PA-5220 after the upgrade to 10.x.  We have a pair of HA PA-5220 in active/passive mode, we never had an auto commit issue before in previous updates, reboots of the firewalls. We have upgraded numerous times before from 8.x all the way to 10.x.  In our recent upgrade to 10.1.x, three of the four firewalls failed on the initial auto commit, of those two of the three eventually finished after retrying a few times in about 10 minutes, one of them though took about 60 minutes to complete.  

I know it's published that it may take 30 minutes for auto commit to finish, but in our case we actually never seen it go over more than 5 minutes in the PA-5220 until the upgrade to 10.x.  When it was failing to auto commit the following error was present on all three firewalls but did eventually cleared by itself. i

configured traffic quota of 0 MB is less than the minimum 32 MB.
Invalid configuration. Please fix errors and try again.
Failed to commit policy to device

in our case this was not service impacting, since it was on HA pair but we do have standalone firewalls that if they are stuck at auto committing then it would be service impacting.

Support basically said this is acceptable/normal.

I just want to know if this is the new normal now for us and to set expectations as such, and what others experience is with auto commit finishing. 

Thanks.

[kiwi]

Hi @RREALICA ,

This could be expected after 10.1.x due to some changes, autocommit fails unless all of the logging disks are ready and have stopped rebuilding.  Expected times for rebuilding the disk will depend on the logging size itself (I've seen it take 60 - 90 minutes).  Allow time for the rebuilding to complete.

Hope this helps,

-Kiwi.

[fwmike2]

Thanks @kiwi.
Experienced a 30 minute reboot to HA ready on PA-5220 going from 10.0.11-h1 to 10.1.6-h6.

Saw the same auto commit errors as @RREALICA.

It did not seem possible to SSH into CLI as admin (to check RAID status) until the system completed the auto commit. 

[codyh]

We also ran into the same issue on an HA Active/Passive pair of PA-5220s. Our upgrade path was 9.1.14-h4 -> 10.0.11-h1 -> 10.1.8. We noticed on the dashboard that HA was down, and also all of the interfaces showed down and not configured. In Tasks we were getting the Auto Commit failure repeatedly with details of:

configured traffic quota of 0 MB is less than the minimum 32 MB.
Invalid configuration. Please fix errors and try again.

Our uptime was 87 minutes when the Auto Commit finally completed. Once that was done everything appeared to be working as expected. Only one firewall in the pair had this happen, the other upgraded in a normal time window (15ish minutes).

As a note, you can check on RAID with cli command: show system raid detail

[fwmike2]

A pair of PA-5250 10.0.11-h1 -> 10.1.6-h6 took ~67 mins per device.

[jkim12]

5410 from 10.2 to 10.2.3 upgrade and enabling HA took 2hr:01 min 

[KTompkins]

Hi Kiwi, could you share what CLI command you ran on the firewall to get the output you show in your screenshot? Thanks in advance.

[kiwi]

Hi @KTompkins ,

The CLI command is :

> show system raid detail

Here's the full KB article: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000wkxPCAQ

Kind regards,

-Kiwi.

[SimonCoggins]

Thanks for including your times. I have 4 x 5220s (2 HA pairs) and managed to hit this on all 4. Your timings assisted in me greatly. Mine were 82mins.

This was going from 9.1.11 -> 10.0.11 -> 10.1.10

[Mohsinh]

I upgrade 5220

It took 45 Mins around for Auto Commit to get completed from failed state till that interface was down. 

Raid System disk was in progress even after interface came back and took 8 hours to get 100% completed

We upgrade from 9.1.6 >> 9.1.11 >> 10.0.11 >> 10.1.10-h1

[stevess]

I'm upgrading an HA pair of 5250's tonight and ran into the RAID rebuild issue on the primary firewall but not the secondary. This is the second time that has been the case, so my theory is that the primary has a much more full disk than the passive firewall (since it rarely runs traffic to fill the logs) so perhaps that is why the passive firewall either finishes too fast to notice or doesn't do the rebuild at all. I'm currently at 98 minutes with no auto-commit completion yet. Its going to be a long night.