Panorama template push fails unless a device group is pushed with it.

[James_Mucklin]

When committing a template only change from panorama to managed firewalls in a HA pair the commit fails. 

When committing a template change along with a device group change it succeeds.

Template only changes commit fine when being pushed down to managed standalone firewalls.

All devices are running PAN-OS 10.1.5-h2

Reviewed the panorama logs along with the logs from the managed firewalls.

From the config daemon logs in Panorama there looks to be an issue with the underlying database.

When a template only commit is pushed, the logs show Panorama failing to obtain operational logs required in the system daemon.

Error messages seen in the logs:

From the configd.log there’s a clear pattern of events;

1.   the commit is pushed from Panorama

2022-05-27 10:26:30.970 +0100 Commit job enqueued. type=2

2022-05-27 10:26:30.973 +0100 start pan_commit_get_cfg_root

2022-05-27 10:26:31.048 +0100 Json array size is 0, nothing will be synced to db

1. The firewall implies there are issues with the HA database objects when it tries to sync;

2022-05-27 10:26:31.048 +0100 Json array size is 0, nothing will be synced to db

2022-05-27 10:26:31.365 +0100 Error:  pan_cfg_get_oplog_from_sysd_obj(pan_cfg_ha_db_sync.c:539): Unable to find the op value in peer.ha.lib.mgmt.impl.usr.base.mdb-oplog; ignoring

2022-05-27 10:26:31.415 +0100 Return detail-ver 10.1.5

2022-05-27 10:26:32.050 +0100 Json array size is 0, nothing will be synced to db

2022-05-27 10:26:32.368 +0100 Error:  pan_cfg_get_oplog_from_sysd_obj(pan_cfg_ha_db_sync.c:539): Unable to find the op value in peer.ha.lib.mgmt.impl.usr.base.mdb-oplog; ignoring

2022-05-27 10:26:32.604 +0100 start pan_cfg_save_commit_candidate

2022-05-27 10:26:33.054 +0100 Json array size is 0, nothing will be synced to db

1. This then fails and reports the failure in the log.

2022-05-27 10:17:09.668 +0100 SEATTLETIME: Time to PROCESSJOB:pan_cfg_commit_to_local_device: 22 secs

2022-05-27 10:17:09.673 +0100 Error:  pan_cfg_replaydb_update_status_by_tids(pan_cfg_replaydb.c:624): pan_cfg_replaydb_update_status_by_tids: List of TIDS is empty

2022-05-27 10:17:09.736 +0100 Json array size is 0, nothing will be synced to db

2022-05-27 10:17:09.841 +0100 Warning:  sc3_sendRegInfo(sc3_register.c:411): SC3R: AK not present.

2022-05-27 10:17:10.049 +0100 client dagger reported op command FAILED

The main error that appears over and over is;

2022-05-27 10:19:00.347 +0100 Error:  pan_cfg_get_oplog_from_sysd_obj(pan_cfg_ha_db_sync.c:539): Unable to find the op value in peer.ha.lib.mgmt.impl.usr.base.mdb-oplog; ignoring

2022-05-27 10:19:01.006 +0100 Json array size is 0, nothing will be synced to db

--------

Now looking at the firewalls themselves, I can see the ‘client’ side of these errors;

2022-05-27 10:20:17.837 +0100 client dagger reported op command FAILED

2022-05-27 10:20:17.982 +0100 client authd reported op command FAILED

2022-05-27 10:20:18.501 +0100 client dagger reported op command FAILED

2022-05-27 10:20:19.460 +0100 client useridd reported op command FAILED

2022-05-27 10:20:19.672 +0100 client useridd reported op command FAILED

2022-05-27 10:20:19.718 +0100 client dagger reported op command FAILED

2022-05-27 10:20:19.720 +0100 client useridd reported op command FAILED

2022-05-27 10:20:19.930 +0100 client authd reported op command FAILED

2022-05-27 10:20:20.524 +0100 client dagger reported op command FAILED

2022-05-27 10:20:21.341 +0100 client dagger reported op command FAILED

2022-05-27 10:20:21.442 +0100 client authd reported op command FAILED

2022-05-27 10:20:21.921 +0100 client dagger reported op command FAILED

2022-05-27 10:20:22.449 +0100 client useridd reported op command FAILED

2022-05-27 10:20:22.646 +0100 client useridd reported op command FAILED

2022-05-27 10:20:22.691 +0100 client useridd reported op command FAILED

At this point, it looks like Panorama is attempting to push the config down the both managed firewalls in the HA pair, but get stopped because of a database syncing issue. But this still doesn’t explain why the commit all seems to work fine when bundled in with a device group push…..

Is this a bug in 10.1.5 ? 

[LAYER_8]

Device group pushes, in general, should be bundled with template updates (when able). 

If there are objects that are referenced in a template, that exist within a device group, and the device group isn't there 'first' or 'with' the commit, we have seen errors before (here). 

[James_Mucklin]

It turns out this was a VM series plugin issue. 

The VM plugins needed to be updated