New Panorama deployment - commit error

[VincentIstas]

Hi Everybody,

We are currently in the process of deploying Panorama and Paloalto firewalls.

We would like to have two layers of device groups:

shared > dg_MGT > dg_MGT-MAR

in such way to have generic and then local firewall rules.

When adding the third level, Panorama generates underneath:

Validation Error:devices -> localhost.localdomain -> device-group -> dg_MGT-MAR 'dg_MGT-MAR' is invalid. meta data not found for dg dg_MGT-MAR

Any idea?

thanks

vincent

[SCantwell_IM]

Can you send us a screen capture of the DeviceGroups from the Panorama tab.  It would be nice to see the hierarchy.
i

[PavelK]

Hello @VincentIstas

would it be also possible to get an output from configuration logs from CLI around the time you get commit error: less mp-log configd.log?

Kind Regards

Pavel 

[VincentIstas]

Hi,

This is the device group hirarchy

and here are the logs:

'cfg.general.dau-purge-interval': NO_MATCHES
/usr/local/bin/iptagsrev_purge.sh: line 71: warning: here-document at line 31 delimited by end-of-file (wanted `EOF')
2023-06-13 20:15:00.392 +0200 logbuffer: no active connection to cms0
2023-06-13 20:15:20.393 +0200 logbuffer: no active connection to cms0
2023-06-13 20:15:33.167 +0200 Error: _send_stats_collection_request(pan_cfg_mgr.c:3876): error getting local log collector for stats collection
2023-06-13 20:15:33.167 +0200 Error: pan_local_cp_es_server_state_update(pan_cfg_mgr.c:3769): error getting local log collector for es server state
2023-06-13 20:15:40.393 +0200 logbuffer: no active connection to cms0
2023-06-13 20:16:00.393 +0200 logbuffer: no active connection to cms0
2023-06-13 20:16:20.393 +0200 logbuffer: no active connection to cms0
2023-06-13 20:16:39.016 +0200 Got authorization info for user "m-istasvi-a": admin role "pa_Admin"; access domain ""; need passwd change 0; passwd expiry days -1; remaining grace period -1; remaining login count -1
2023-06-13 20:16:40.393 +0200 logbuffer: no active connection to cms0
2023-06-13 20:16:42.649 +0200 Got authorization info for user "m-istasvi-a": admin role "pa_Admin"; access domain ""; need passwd change 0; passwd expiry days -1; remaining grace period -1; remaining login count -1
2023-06-13 20:17:00.394 +0200 logbuffer: no active connection to cms0
Connection to Update server: updates.paloaltonetworks.com completed successfully, initiated by 10.12.68.148
2023-06-13 20:17:19.282 +0200 Error: pan_cfg_get_session_by_cookie(pan_cfg_mgr.c:14897): session with cookie doesn't exist2023-06-13 20:17:20.394 +0200 logbuffer: no active connection to cms0
2023-06-13 20:17:29.971 +0200 Got authorization info for user "m-istasvi-a": admin role "pa_Admin"; access domain ""; need passwd change 0; passwd expiry days -1; remaining grace period -1; remaining login count -1
2023-06-13 20:17:30.429 +0200 Error: pan_cfg_get_platform_mac(pan_cfg_utils.c:10645): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac failed (errno=9)
2023-06-13 20:17:30.558 +0200 Error: pan_cfg_get_platform_mac_count(pan_cfg_utils.c:10683): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac-count failed (errno=9)
2023-06-13 20:17:35.005 +0200 Error: _pan_schema_verify_node(pan_schema_obj.c:7938): is unexpected , node: cluster near line 1
2023-06-13 20:17:35.191 +0200 client authd reported op command FAILED
2023-06-13 20:17:36.305 +0200 client dagger reported op command FAILED
2023-06-13 20:17:40.394 +0200 logbuffer: no active connection to cms0
2023-06-13 20:18:00.394 +0200 logbuffer: no active connection to cms0
2023-06-13 20:18:20.394 +0200 logbuffer: no active connection to cms0
2023-06-13 20:18:38.079 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='business-systems']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.168 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='collaboration']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.257 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='general-internet']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.345 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='media']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.430 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='networking']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.517 +0200 Error: pan_cfg_get_predef_hash_apps(pan_cfg_cloudupdate.c:874): No completions found for predefined/application-type/category/entry[@name='saas']/subcategory/entry/@name in pred__hash
2023-06-13 20:18:38.587 +0200 Error: pan_cfg_get_sctp_field_values(pan_apptypedb.c:3077): No mapping hash for type 1
2023-06-13 20:18:40.395 +0200 logbuffer: no active connection to cms0
2023-06-13 20:19:00.395 +0200 logbuffer: no active connection to cms0
2023-06-13 20:19:19.717 +0200 Error: pan_cfg_get_platform_mac(pan_cfg_utils.c:10645): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac failed (errno=9)
2023-06-13 20:19:19.847 +0200 Error: pan_cfg_get_platform_mac_count(pan_cfg_utils.c:10683): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac-count failed (errno=9)
2023-06-13 20:19:20.395 +0200 logbuffer: no active connection to cms0
2023-06-13 20:19:20.671 +0200 panorama license check from UI. Panorama Device Management license is valid.
2023-06-13 20:19:21.573 +0200 devtelem: UI final resp: <deviceTelemetry>
<region-list>
</region-list>
</deviceTelemetry>

2023-06-13 20:19:40.395 +0200 logbuffer: no active connection to cms0
2023-06-13 20:19:51.435 +0200 get_tlolevel_xpath
2023-06-13 20:19:51.435 +0200 New xpath is /config/devices/entry[@name='localhost.localdomain']/deviceconfig/system/device-telemetry
2023-06-13 20:19:51.436 +0200 String is <device-telemetry>
<threat-prevention>yes</threat-prevention>
<device-health-performance>yes</device-health-performance>
<product-usage>yes</product-usage>
</device-telemetry>
2023-06-13 20:19:51.436 +0200 Warning: pan_hash_init(pan_hash.c:113): nbuckets 16536 is not power of 2!
2023-06-13 20:19:51.436 +0200 Warning: pan_hash_init(pan_hash.c:113): nbuckets 16535 is not power of 2!
2023-06-13 20:19:51.436 +0200 Error: _pan_schema_verify_node(pan_schema_obj.c:7771): is invalid , node: region near line 1
2023-06-13 20:19:51.436 +0200 Error: pan_cfg_engine_execute_request(pan_cfg_engine.c:4540): processing command edit failed
2023-06-13 20:19:51.882 +0200 Error: pan_cfg_mgr_handle_local_request(pan_cfg_mgr.c:40752): The following request could not be handled:<request cmd="edit" obj="/config/devices/entry[@name='localhost.localdomain']/deviceconfig/system/device-telemetry" cookie="1622704964961284"><device-telemetry><device-health-performance>yes</device-health-performance><product-usage>yes</product-usage><threat-prevention>yes</threat-prevention><region/></device-telemetry></request>'cfg.general.dau-purge-interval': NO_MATCHES
/usr/local/bin/iptagsrev_purge.sh: line 71: warning: here-document at line 31 delimited by end-of-file (wanted `EOF')
2023-06-13 20:20:00.396 +0200 logbuffer: no active connection to cms0
2023-06-13 20:20:07.136 +0200 Plugin: set user for vmware_vcenter
2023-06-13 20:20:07.137 +0200 User is set for plugin vmware_vcenter
2023-06-13 20:20:07.137 +0200 Plugin: set user for cisco
2023-06-13 20:20:07.138 +0200 User is set for plugin cisco
2023-06-13 20:20:07.504 +0200 Commit job enqueued. type=2
2023-06-13 20:20:07.508 +0200 start pan_commit_get_cfg_root
2023-06-13 20:20:07.943 +0200 Return detail-ver 11.0.1
2023-06-13 20:20:08.055 +0200 SEATTLETIME: Time to COMMITCFGCOOKIE:pan_xmlCopyNode predefined, panoramanode: 1 secs
2023-06-13 20:20:08.205 +0200 start pan_cfg_save_commit_candidate
2023-06-13 20:20:08.993 +0200 COMMIT SaveCandidate: Time to unlink devices node: 0 secs
2023-06-13 20:20:09.434 +0200 COMMIT SaveCandidate: Time to merge global and custom global: 1 secs
2023-06-13 20:20:09.434 +0200 COMMIT SaveCandidate: Time to make other node copies: 0 secs
2023-06-13 20:20:09.434 +0200 COMMIT SaveCandidate: Time to add nodes back: 0 secs
2023-06-13 20:20:09.968 +0200 COMMIT SaveCandidate: Time to commit transform: 0 secs
2023-06-13 20:20:09.968 +0200 Saving candidate to .candidate-snapshot.xml.128
2023-06-13 20:20:09.968 +0200 /tmp/.iddone not there when writing to /opt/pancfg/mgmt/commit-candidates/.candidate-snapshot.xml.128
2023-06-13 20:20:10.105 +0200 COMMIT SaveCandidate: Time to save .candidate.xml: 1 secs
2023-06-13 20:20:10.157 +0200 COMMIT commit: Time to pan_cfg_save_candidate_config: 2 secs
2023-06-13 20:20:10.868 +0200 done pan_cfg_save_commit_candidate
2023-06-13 20:20:10.868 +0200 SEATTLETIME: Time to GENCOMMITCAND:pan_cfg_save_commit_candidate: 2 secs
2023-06-13 20:20:10.868 +0200 SEATTLETIME: Time to COMMITCFGCOOKIE:pan_cfg_generate_commit_candidates: 2 secs
2023-06-13 20:20:10.975 +0200 SEATTLETIME: Time to PRECOMMIT:pan_cfg_commit_cfg_by_cookie: 3 secs
2023-06-13 20:20:10.975 +0200 Takes 3 seconds to generate commit candidate in cfg_by_cookie.
2023-06-13 20:20:10.975 +0200 SEATTLETIME: Time to PROCESSJOB:pan_cfg_pre_commit_processing: 3 secs
2023-06-13 20:20:11.055 +0200 Start to merge template-stack config
2023-06-13 20:20:11.059 +0200 Done to merge template-stack config
2023-06-13 20:20:11.059 +0200 SEATTLETIME: Time to PROCESSJOB:pan_cfg_commit_merge_tplstacks_config: 1 secs
2023-06-13 20:20:11.230 +0200 Start plugin pre commit
2023-06-13 20:20:11.695 +0200 Error: pan_xml_transform_node_by_ssfile_inmemory(pan_xml_utils.c:765): transform file /opt/plugins/installed/vm_series/xsl/vm-series.xsl does not exist
2023-06-13 20:20:11.695 +0200 Error: pan_cfg_transform_fullpath(pan_cfg_utils.c:6950): error generating transform /opt/plugins/installed/vm_series/xsl/vm-series.xsl
2023-06-13 20:20:12.217 +0200 Error: pan_cfg_transform_fullpath(pan_cfg_utils.c:6950): error generating transform /opt/plugins/xsl/input-plugins.xsl
/proc/cpuinfo: No such file or directory
/proc/cpuinfo: No such file or directory
2023-06-13 20:20:18.011 +0200 Done plugin pre commit
2023-06-13 20:20:18.011 +0200 SEATTLETIME: Time to PROCESSJOB:pan_plugin_pre_commit_process: 7 secs
2023-06-13 20:20:18.189 +0200 number of modified TPL: 2
2023-06-13 20:20:18.189 +0200 Find dg for template tpl_MGT
2023-06-13 20:20:18.189 +0200 Find dg for template tpl-stack_MGT
2023-06-13 20:20:18.189 +0200 number of modified DG: 2
2023-06-13 20:20:18.311 +0200 Verifying Configuration
2023-06-13 20:20:18.311 +0200 Warning: pan_hash_init(pan_hash.c:113): nbuckets 16535 is not power of 2!
2023-06-13 20:20:18.314 +0200 024201002452's auto-push flag has been updated.
2023-06-13 20:20:18.321 +0200 Error: pan_cfg_dgname_validate(pan_cfg_devicegroups.c:2303): meta data not found for dg dg_MGT-MAR
2023-06-13 20:20:18.321 +0200 Error: pan_schema_verify_attribute(pan_schema_types.c:1054): 'dg_MGT-MAR' is invalid. meta data not found for dg dg_MGT-MAR near line 0
2023-06-13 20:20:18.321 +0200 Error: pan_schema_verify_attr(pan_schema_obj.c:5783): attribute name breaks schema at line 0
2023-06-13 20:20:18.334 +0200 Error: pan_cfg_verify_ex(pan_cfg_commit_handler.c:2992): invalid configuration. Schema verification failed.
2023-06-13 20:20:18.334 +0200 Clearing commit completion cache
2023-06-13 20:20:18.334 +0200 Error: pan_jobmgr_process_job(pan_job_mgr.c:3952): error verifying commit candidate
2023-06-13 20:20:18.445 +0200 Removing /tmp/.iddone in pan_cfg_remove_temporary_files
2023-06-13 20:20:18.445 +0200 Removing /opt/pancfg/mgmt/saved-configs/.revertible.candidate-snapshot.xml in pan_cfg_remove_temporary_files
2023-06-13 20:20:18.450 +0200 Schema validation including uuid check for job 514 takes 0 seconds
2023-06-13 20:20:18.626 +0200 Warning: sc3_sendRegInfo(sc3_register.c:424): SC3R: AK not present.
2023-06-13 20:20:19.042 +0200 Error: pan_cfg_get_platform_mac(pan_cfg_utils.c:10645): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac failed (errno=9)
2023-06-13 20:20:19.172 +0200 Error: pan_cfg_get_platform_mac_count(pan_cfg_utils.c:10683): pan_sys_exec2(): /usr/local/bin/sdb -n cfg.platform.mac-count failed (errno=9)
curl: (7) Failed to connect to localhost port 9200: Connection refused
2023-06-13 20:20:19.351 +0200 Error: pan_cfg_get_cms_msg(pan_cfg_mgr.c:45135): Failed to get es_status health
2023-06-13 20:20:19.351 +0200 Error: pan_interlc_check_conn_status(pan_cfg_load_ring_handler.c:518): error getting group in local ring
2023-06-13 20:20:19.354 +0200 Destructing config_vars
2023-06-13 20:20:19.355 +0200 Setting commitvars destruct done flag
2023-06-13 20:20:19.515 +0200 Done Destructing config_vars
2023-06-13 20:20:20.396 +0200 logbuffer: no active connection to cms0
2023-06-13 20:20:33.209 +0200 Error: _send_stats_collection_request(pan_cfg_mgr.c:3876): error getting local log collector for stats collection
2023-06-13 20:20:33.209 +0200 Error: pan_local_cp_es_server_state_update(pan_cfg_mgr.c:3769): error getting local log collector for es server state
2023-06-13 20:20:40.396 +0200 logbuffer: no active connection to cms0

[PavelK]

Hello @VincentIstas

thank you for reply.

From the logs, it looks like that below is breaking the configuration:

2023-06-13 20:20:18.321 +0200 Error: pan_schema_verify_attr(pan_schema_obj.c:5783): attribute name breaks schema at line 0
2023-06-13 20:20:18.334 +0200 Error: pan_cfg_verify_ex(pan_cfg_commit_handler.c:2992): invalid configuration. Schema verification failed.

I am not clear what "attribute name at line 0" indicates. I would recommend general troubleshooting steps from this KB, please refer to the section: Revert the candidate configuration. After you revert it, please try to create the device group: dg_MGT-MAR, then commit, then add Firewall to device group and commit,... with this you will know what specific step is breaking configuration.

Alternatively, delete the device group dg_MGT-MAR and its associated configuration and then try to add single step at a time and commit it until you either hit the error or complete the configuration.

Kind Regards

Pavel

[aquariusgrapefruit · Accepted solution]

I realize this might be an inactive thread, but it was only of the two search results for the "meta data not found for dg" error. I wanted to add what happened to work for me after reading through this discussion and reviewing my configd.log. Thank you to VincentIstas and PavelK for sharing their experiences and troubleshooting steps. For reference, I am currently using the Panorama 11.0.x release. (Other search results mention similar issues on the 10.2.x release but only for Prisma.)

I tried deleting and re-creating my new device group multiple times, adjusting/editing it in different ways, putting it under different existing (working) device groups, and committing at different stages to try and isolate the issue. None of that worked.

What did work, after noticing some other strange issues in Panorama, was to commit all changes to Panorama rather than only trying to commit my changes. This cleared the issue immediately, even though nobody else on our team had any pending changes in Panorama. Hopefully this additional tip helps someone else who finds this thread, in the event PavelK's troubleshooting steps don't reveal another cause.

[PaulArcellx]

My resolution was similar to @aquariusgrapefruit. I was hitting the "meta data not found for dg" error when trying to rename my device groups. I was making this change as a SAML superuser, but that SAML superuser was unable to do a full commit. I had to log in as a local superuser to make the changes as a full commit instead.

[NickCostanzo]

Had the same experiences here running 11.1.0 Panorama. Committing all changes pushed it through. I still am not able to connect it to Panorama but I suspect I might've missed a step on this.

[Brandon_Wertz]

I'm not exactly certain if this relates to everyone's issue here, but I think the commit failures have something to do with a database schema change when Panorama is upgraded to PANOS 10.2.X.  I could be wrong but I heard that the database schema changed from 10.1.X to 10.2.X.  It's this schema change that is creating the inability to do "partial" commits and is requiring full commit.

If this is in-fact the problem I don't know if it goes away if managed firewalls are equal to or above PANOS 10.2+

[Majid-Yaseen]

I am having the same issue on Panorama 11.0.2-h2 so it does not go away for later versions 

[Majid-Yaseen]

Commit All Changes worked for me also 
It would be good to understand why it gives the error when only commiting changes Made By an admin as this is the process that most NGFW admin's use

[NickCostanzo]

@Majid-Yaseen  Are the firewalls you manage 10.2.x and above? All of mine are 10.1.x or 10.0.x at this time where I have the issues.

[afalfaro]

Without a doubt the answer given by @aquariusgrapefruit  was the one that worked for me. Apply commit for everything and not just my user.

The version of the panorama is 10.2.8 and the version of the devices was 10.1.11.