PanOS 11.1.0 Upgrade - Panorama Refuses to Commit or Push on a Multi-VSYS System

[nccbrettsmith]

Hey Team,

Has anyone encountered any problems performing the PanOS 11.1.0 Upgrade? I've encountered the following issue after an upgrade, where PanOS (Panorama) would not commit changes, much less push them to our devices. The configd.log file shows the following:

2023-12-09 16:36:16.778 +1100 DG-push(selective): Waiting for DG file to be written for XXXX
2023-12-09 16:36:16.867 +1100 Error: pan_populate_mvsys_policy(pan_cfg_dg_tpl_utils.c:8032): File /opt/pancfg/mgmt/groups/XXXX/panorama-selective-mvsys-config.xml does not exist, aborting
2023-12-09 16:36:16.867 +1100 Error: pan_cfg_generate_multidg_push_or_diffall_msg_for_device(pan_cfg_shared_policy.c:3980): Failed to populate policy node for XXXX
2023-12-09 16:36:16.867 +1100 Error: pan_cfg_sp_push(pan_cfg_shared_policy.c:5514): error generating push/diffall request to XXXXXX
2023-12-09 16:36:16.873 +1100 Error: pan_populate_mvsys_policy(pan_cfg_dg_tpl_utils.c:8032): File /opt/pancfg/mgmt/groups/XXXX/panorama-selective-mvsys-config.xml does not exist, aborting
2023-12-09 16:36:16.873 +1100 Error: pan_cfg_generate_multidg_push_or_diffall_msg_for_device(pan_cfg_shared_policy.c:3980): Failed to populate policy node for XXXX
2023-12-09 16:36:16.873 +1100 Error: pan_cfg_sp_push(pan_cfg_shared_policy.c:5514): error generating push/diffall request to XXXXXX
2023-12-09 16:36:16.927 +1100 DG-push(selective): Waiting for DG file to be written for XXXX

It looks to me like an upgrade migration process didn't work when we moved from PanOS 10.2.7 (we did a multi-hop upgrade, but it was working at this step as far as we knew as we did changes to GlobalProtect configuration at this point).

[S.Cantwell]

Hello

Instead of a selective push, why are you not doing a full commit?
i have seen generally, errors, when the PANOS needs a full commit (vs selective) and fails/errors when it is not done.

Maybe be a CLI command of "commit force" to see if that helps.

[nccbrettsmith]

Hi There,

In the end it was found to be a bug within version 10.2.5 and its migration of our configuration. The panorama creates a "default" log collection profile again during the upgrade and this cannot be committed as the firewalls also come out of the box with the uprade with a "default" collection profile. To resolve, you must rename one or both of these configurations to allow the commit to succeed. A selective commit did not work, and additionally, the following setting must be changed:

Select Panorama > Setup > Management and edit the Panorama Settings to enabled Shared Unused Address and Service Objects with Devices.

Once this is done, then the commit will be attempted with an error displayed about the conflict. Resolve the conflict and you can get on with the upgrade.

References: PAN-OS 10.2.5 Known Issues (paloaltonetworks.com)

See PAN-225337

Thankyou for advising of the commit force functionality via the CLI however.