Failed commiting config from Panorama

Reply
Highlighted
L4 Transporter

Hi,

 

This is job id output. Nothing significant. This FW has the license attached in Panorama, but if we go to support in this PA we cant see the support license, but we can see URL and TP license. I dont know if this FW hasnt license support in ownself it could be cause this problem........

 

> show jobs id 34971

Enqueued Dequeued ID Type Status Result Completed
------------------------------------------------------------------------------------------------------------------------------
2018/01/16 23:12:02 23:12:02 34971 CommitAll FIN FAIL 23:12:18
Warnings:
Details:

Highlighted
L4 Transporter

This is the ms.log output. Commit was failed at 11:12:00. What is the exact error for this failed ?

 

2018-01-17 11:07:35.048 +0100 ** generating report for time from 1516180055 to 1516183654
2018-01-17 11:10:02.298 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:10:02.298 +0100 Error: _pan_mgmtop_upload_handler(pan_ops_common.c:21996): Failed to purge old uploaded files
2018-01-17 11:10:03.330 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:10:03.330 +0100 Error: _pan_mgmtop_upload_handler(pan_ops_common.c:21996): Failed to purge old uploaded files
2018-01-17 11:10:42.824 +0100 dnscfgmod: FQDN Refresh: Periodic TTL Expiry Refresh
2018-01-17 11:10:42.824 +0100 dnscfgmod: Main refresh function: (TTL Expiry)
2018-01-17 11:10:42.825 +0100 dnscfgmod:Fqdn refresh job 35104 scheduled
2018-01-17 11:10:42.825 +0100 FqdnRefresh job started processing. Dequeue time=2018/01/17 11:10:42 2018-01-17 11:11:43.364 +0100 dnscfgmod: Resolving fqdns took 61 secs
2018-01-17 11:11:43.364 +0100 dnscfgmod: No IP changes seen after resolving FQDNS. Skipping config push to device.
2018-01-17 11:12:15.426 +0100 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:5534): failed to fetch: NO_MATCHES
2018-01-17 11:12:16.366 +0100 CommitAll job started processing. Dequeue time=2018/01/17 11:12:16. JobId=35105.User: Panorama-dimjmt
2018-01-17 11:12:18.072 +0100 Panorama push template EEXTERNAS with merge-with-candidate-cfg flags set.JobId=35105.User=Panorama-dimjmt. Dequeue time=2018/01/17 11:12:16.
2018-01-17 11:12:18.075 +0100 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:5534): failed to fetch: NO_MATCHES
2018-01-17 11:12:18.091 +0100 Error: pan_cfg_transform_fullpath(pan_cfg_utils.c:5848): error generating transform /opt/pancfg/mgmt/factory/tplrenamemapfrompushreq.xsl
2018-01-17 11:12:18.091 +0100 Error: pan_cfg_tpl_renamemap_from_request(pan_cfg_templates.c:3367): failed to generate tpl rename map from request
2018-01-17 11:12:18.091 +0100 no rename map in request
2018-01-17 11:12:18.170 +0100 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:5509): failed to fetch: NO_MATCHES
2018-01-17 11:12:18.411 +0100 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:5509): failed to fetch: NO_MATCHES
2018-01-17 11:12:29.337 +0100 Error: pan_cfg_sp_generate_candidate_vsys_sps_by_root(pan_cfg_shared_policy.c:4755): no policy node under push request
2018-01-17 11:12:29.509 +0100 Error: pan_cfg_sp_generate_candidate_vsys_sps_by_root(pan_cfg_shared_policy.c:4755): no policy node under push request
2018-01-17 11:12:29.510 +0100 detail : Commit from Panorama. Merged with candidate config: Yes. Commit parameters: force=false, device_network=true, shared_object=true. Commit All Vsys.
2018-01-17 11:12:29.532 +0100 Created Verify Thread
2018-01-17 11:12:29.534 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.534 +0100 Error: pan_cfg_commit_to_local_device(pan_cfg_commit_handler.c:2823): Failed to move and rename candidate xml file
2018-01-17 11:12:29.536 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.537 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.567 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.568 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.607 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:29.633 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:30.548 +0100 Got HA info from sysd: Local_state: unknown, Peer_state: unknown2018-01-17 11:12:32.299 +0100 Verifying Configuration
2018-01-17 11:12:33.940 +0100 Takes 1 seconds to verify schema.
2018-01-17 11:12:33.940 +0100 Clearing commit completion cache2018-01-17 11:12:34.666 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:35.141 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:12:36.717 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:12:36.718 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:12:36.718 +0100 ** generating report for time from 1516180356 to 1516183955
2018-01-17 11:15:01.165 +0100 pan_dynupdsch_local_refresh(pan_cfg_dynupdsch.c:2032): scheduled-update: "_SystemWildfireUpdate_" refreshing of WildFire
2018-01-17 11:15:01.189 +0100 API Key is not set in cryptod
2018-01-17 11:15:01.435 +0100 Checking to purge appstatdb logtype
2018-01-17 11:15:02.110 +0100 API Key is not set in cryptod
2018-01-17 11:15:02.110 +0100 Error: pan_support_get_info(pan_ops_common.c:9700): Error executing/reading output of command grep -v "^#" /etc/pan_upd.conf | head -1 | awk '{print $2}'
2018-01-17 11:15:02.111 +0100 Error: __pan_sys_system_cb(pan_sys.c:801): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:15:02.111 +0100 updater error code:-1
2018-01-17 11:15:02.111 +0100 Error: check_content_upgrade_info(pan_ops_content.c:2894): Failed to check Antivirus content upgrade info due to generic communication error
2018-01-17 11:15:02.111 +0100 updater error code:-1
2018-01-17 11:15:02.112 +0100 No new Antivirus updates available for download
'cfg.fail-conn-on-cert': NO_MATCHES
NO_MATCHES
NO_MATCHES
2018-01-17 11:16:16.802 +0100 API Key is not set in cryptod
'cfg.fail-conn-on-cert': NO_MATCHES
NO_MATCHES
NO_MATCHES
2018-01-17 11:17:13.469 +0100 File successfully downloaded
2018-01-17 11:17:13.469 +0100 File '/opt/pancfg/mgmt/wildfire-images/panupv2-all-wildfire-210236-212598.tgz' successfully downloaded for post_proc_cont.
2018-01-17 11:17:13.559 +0100 WildFire job started processing. Dequeue time=2018/01/17 11:17:13 2018-01-17 11:17:32.131 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.131 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.132 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.205 +0100 Warning: pan_sigdb_get_idsev_map(pan_sigdb.c:936): /opt/pancfg/mgmt/global/wpc.xml.sev doesn't exist
2018-01-17 11:17:32.205 +0100 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1470): failed to get wpc idsev map
2018-01-17 11:17:32.205 +0100 Warning: pan_sigdb_get_wpcdb(pan_sigdb.c:1098): /opt/pancfg/mgmt/global/wpc.xml.db doesn't exist
2018-01-17 11:17:32.205 +0100 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1474): failed to get wpcinfo db
2018-01-17 11:17:34.937 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:34.952 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:17:39.122 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:17:39.123 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:17:39.123 +0100 ** generating report for time from 1516180659 to 1516184258
2018-01-17 11:17:39.127 +0100 Update logforward config, flags: mdata[1], log setting[0]
2018-01-17 11:17:39.129 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:39.129 +0100 Error: pan_mgmtop_sync_content_to_peer(pan_ops_content.c:4059): failed to remove old content files
2018-01-17 11:17:45.114 +0100 client device reported Phase 1 was SUCCESSFUL
2018-01-17 11:17:45.115 +0100 Error: pan_get_current_gp_datafile_version(pan_cfg_utils.c:5387): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:45.144 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:45.144 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:45.145 +0100 removing the content of /opt/pancfg/mgmt/wf_ramdisk/updates/oldwildfire
2018-01-17 11:17:45.245 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:45.267 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:46.176 +0100 Got HA info from sysd: Local_state: unknown, Peer_state: unknown2018-01-17 11:19:13.152 +0100 Getting authorization info for user admin failed.
2018-01-17 11:19:13.161 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:19:19.289 +0100 client authd reported op command was SUCCESSFUL
2018-01-17 11:19:31.380 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:19:31.395 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:19:33.317 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:19:33.317 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:19:33.318 +0100 ** generating report for time from 1516180773 to 1516184372

Community Team Member

Hi @soporteseguridad,

 

What is the PAN-OS version ?

 

fork() failed! errno=12 (Cannot allocate memory)

 

That might indicate a memory leak issue.

 

From the PAN-OS 8.0.6 release notes :

PAN-86353 - Fixed an issue on the Panorama management server where combinations of reports and log queries intermittently produced a slow memory leak that causes memory‐related errors such as commit failures.

 

Hope this helps !

Cheers !

-Kiwi.

Highlighted
L4 Transporter

PAN-OS is 8.0.3. But the commit failed occured in FW, not only running this commit from panorama. 

Highlighted
Community Team Member

Hi @soporteseguridad,

 

In that case you might be hitting this :

 

From the PAN-OS 8.0.6 release notes : 

PAN-81100 - Fixed an issue on the firewall and Panorama management server where a memory leak caused several operations to fail, such as commits, FQDN refreshes, and content updates

 

Eitherway, based on the logs it looks like a memory issue so you might want to check the memory usage on the device and look for a process that might be the culprit ... a restart of that process might be a workaround for you in the meantime.

 

To check resource usage you can use the following command :

 

> show system resources follow

 

Cheers !

-Kiwi,

View solution in original post

Highlighted
L4 Transporter

Yes, we will try a restart and if it stills happen procced to upgrade. Thanks a lot kiwi

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!