Failed commiting config from Panorama

[soporteseguridad]

Hi,

We have a Panorama with several FWs managed. We commited the config but in one of these FWs was failed.

Looking in panorama we see that this device is out of sync (in templates and shred policy). how can i force this commit?? or to have any reason for this fail??.  I dont see any error or how to investigate....

[nawaza]

Why did the commit fail - what does the exact reason for commit failure read?

Ajaz Nawaz

JNCIE-SEC No.254

CCIE-RS No.15721

[soporteseguridad]

I will try to check again. But in commit error in Panorama we didnt see any cause

[soporteseguridad]

ç

I tried to do commit in Panorama but it failed again. I attach the screeshots. I cant see any cause for this failed. Any idea? 

[Mick_Ball]

if you connect to one of the devices that the commit was sent to you can select "Tasks" on bottom RH corner of screen.

select the  commit and this will give you more detail.

[soporteseguridad]

We dont see any error:

[Mick_Ball]

are you looking from panorama or the palo alto firewall.

my suggestion was to go to the firewall itself to review the warning.

[BPry]

@soporteseguridad,

I really won't rely on Panorama to give you the same information that the device would in this instance. I would take @Mick_Ball's suggestion and actually look directly at the firewall, it should give you an indication on why the commit is failing. 

[soporteseguridad]

I see this in monitor logs.

The commit was done last night so i can see this commit in task in order to do "show jobs id". ANy log file where i can see the cause for this commit error??

[BPry]

@soporteseguridad,

The 'show jobs id id' would give you all the warnings, details, and description associated with the commit. 

[soporteseguridad]

Hi,

This is job id output. Nothing significant. This FW has the license attached in Panorama, but if we go to support in this PA we cant see the support license, but we can see URL and TP license. I dont know if this FW hasnt license support in ownself it could be cause this problem........

> show jobs id 34971

Enqueued Dequeued ID Type Status Result Completed
------------------------------------------------------------------------------------------------------------------------------
2018/01/16 23:12:02 23:12:02 34971 CommitAll FIN FAIL 23:12:18
Warnings:
Details:

[soporteseguridad]

This is the ms.log output. Commit was failed at 11:12:00. What is the exact error for this failed ?

2018-01-17 11:07:35.048 +0100 ** generating report for time from 1516180055 to 1516183654
2018-01-17 11:10:02.298 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:10:02.298 +0100 Error: _pan_mgmtop_upload_handler(pan_ops_common.c:21996): Failed to purge old uploaded files
2018-01-17 11:10:03.330 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:10:03.330 +0100 Error: _pan_mgmtop_upload_handler(pan_ops_common.c:21996): Failed to purge old uploaded files
2018-01-17 11:10:42.824 +0100 dnscfgmod: FQDN Refresh: Periodic TTL Expiry Refresh
2018-01-17 11:10:42.824 +0100 dnscfgmod: Main refresh function: (TTL Expiry)
2018-01-17 11:10:42.825 +0100 dnscfgmod:Fqdn refresh job 35104 scheduled
2018-01-17 11:10:42.825 +0100 FqdnRefresh job started processing. Dequeue time=2018/01/17 11:10:42 2018-01-17 11:11:43.364 +0100 dnscfgmod: Resolving fqdns took 61 secs
2018-01-17 11:11:43.364 +0100 dnscfgmod: No IP changes seen after resolving FQDNS. Skipping config push to device.
2018-01-17 11:12:15.426 +0100 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:5534): failed to fetch: NO_MATCHES
2018-01-17 11:12:16.366 +0100 CommitAll job started processing. Dequeue time=2018/01/17 11:12:16. JobId=35105.User: Panorama-dimjmt
2018-01-17 11:12:18.072 +0100 Panorama push template EEXTERNAS with merge-with-candidate-cfg flags set.JobId=35105.User=Panorama-dimjmt. Dequeue time=2018/01/17 11:12:16.
2018-01-17 11:12:18.075 +0100 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:5534): failed to fetch: NO_MATCHES
2018-01-17 11:12:18.091 +0100 Error: pan_cfg_transform_fullpath(pan_cfg_utils.c:5848): error generating transform /opt/pancfg/mgmt/factory/tplrenamemapfrompushreq.xsl
2018-01-17 11:12:18.091 +0100 Error: pan_cfg_tpl_renamemap_from_request(pan_cfg_templates.c:3367): failed to generate tpl rename map from request
2018-01-17 11:12:18.091 +0100 no rename map in request
2018-01-17 11:12:18.170 +0100 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:5509): failed to fetch: NO_MATCHES
2018-01-17 11:12:18.411 +0100 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:5509): failed to fetch: NO_MATCHES
2018-01-17 11:12:29.337 +0100 Error: pan_cfg_sp_generate_candidate_vsys_sps_by_root(pan_cfg_shared_policy.c:4755): no policy node under push request
2018-01-17 11:12:29.509 +0100 Error: pan_cfg_sp_generate_candidate_vsys_sps_by_root(pan_cfg_shared_policy.c:4755): no policy node under push request
2018-01-17 11:12:29.510 +0100 detail : Commit from Panorama. Merged with candidate config: Yes. Commit parameters: force=false, device_network=true, shared_object=true. Commit All Vsys.
2018-01-17 11:12:29.532 +0100 Created Verify Thread
2018-01-17 11:12:29.534 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.534 +0100 Error: pan_cfg_commit_to_local_device(pan_cfg_commit_handler.c:2823): Failed to move and rename candidate xml file
2018-01-17 11:12:29.536 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.537 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.567 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.568 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:12:29.607 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:29.633 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:30.548 +0100 Got HA info from sysd: Local_state: unknown, Peer_state: unknown2018-01-17 11:12:32.299 +0100 Verifying Configuration
2018-01-17 11:12:33.940 +0100 Takes 1 seconds to verify schema.
2018-01-17 11:12:33.940 +0100 Clearing commit completion cache2018-01-17 11:12:34.666 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:12:35.141 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:12:36.717 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:12:36.718 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:12:36.718 +0100 ** generating report for time from 1516180356 to 1516183955
2018-01-17 11:15:01.165 +0100 pan_dynupdsch_local_refresh(pan_cfg_dynupdsch.c:2032): scheduled-update: "_SystemWildfireUpdate_" refreshing of WildFire
2018-01-17 11:15:01.189 +0100 API Key is not set in cryptod
2018-01-17 11:15:01.435 +0100 Checking to purge appstatdb logtype
2018-01-17 11:15:02.110 +0100 API Key is not set in cryptod
2018-01-17 11:15:02.110 +0100 Error: pan_support_get_info(pan_ops_common.c:9700): Error executing/reading output of command grep -v "^#" /etc/pan_upd.conf | head -1 | awk '{print $2}'
2018-01-17 11:15:02.111 +0100 Error: __pan_sys_system_cb(pan_sys.c:801): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:15:02.111 +0100 updater error code:-1
2018-01-17 11:15:02.111 +0100 Error: check_content_upgrade_info(pan_ops_content.c:2894): Failed to check Antivirus content upgrade info due to generic communication error
2018-01-17 11:15:02.111 +0100 updater error code:-1
2018-01-17 11:15:02.112 +0100 No new Antivirus updates available for download
'cfg.fail-conn-on-cert': NO_MATCHES
NO_MATCHES
NO_MATCHES
2018-01-17 11:16:16.802 +0100 API Key is not set in cryptod
'cfg.fail-conn-on-cert': NO_MATCHES
NO_MATCHES
NO_MATCHES
2018-01-17 11:17:13.469 +0100 File successfully downloaded
2018-01-17 11:17:13.469 +0100 File '/opt/pancfg/mgmt/wildfire-images/panupv2-all-wildfire-210236-212598.tgz' successfully downloaded for post_proc_cont.
2018-01-17 11:17:13.559 +0100 WildFire job started processing. Dequeue time=2018/01/17 11:17:13 2018-01-17 11:17:32.131 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.131 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.132 +0100 Warning: pan_hash_init(pan_hash.c:112): nbuckets 10000 is not power of 2!
2018-01-17 11:17:32.205 +0100 Warning: pan_sigdb_get_idsev_map(pan_sigdb.c:936): /opt/pancfg/mgmt/global/wpc.xml.sev doesn't exist
2018-01-17 11:17:32.205 +0100 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1470): failed to get wpc idsev map
2018-01-17 11:17:32.205 +0100 Warning: pan_sigdb_get_wpcdb(pan_sigdb.c:1098): /opt/pancfg/mgmt/global/wpc.xml.db doesn't exist
2018-01-17 11:17:32.205 +0100 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1474): failed to get wpcinfo db
2018-01-17 11:17:34.937 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:34.952 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:17:39.122 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:17:39.123 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:17:39.123 +0100 ** generating report for time from 1516180659 to 1516184258
2018-01-17 11:17:39.127 +0100 Update logforward config, flags: mdata[1], log setting[0]
2018-01-17 11:17:39.129 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:39.129 +0100 Error: pan_mgmtop_sync_content_to_peer(pan_ops_content.c:4059): failed to remove old content files
2018-01-17 11:17:45.114 +0100 client device reported Phase 1 was SUCCESSFUL
2018-01-17 11:17:45.115 +0100 Error: pan_get_current_gp_datafile_version(pan_cfg_utils.c:5387): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:45.144 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:45.144 +0100 Error: pan_sys_system_int(pan_sys.c:563): fork() failed! errno=12 (Cannot allocate memory)2018-01-17 11:17:45.145 +0100 removing the content of /opt/pancfg/mgmt/wf_ramdisk/updates/oldwildfire
2018-01-17 11:17:45.245 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:45.267 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:17:46.176 +0100 Got HA info from sysd: Local_state: unknown, Peer_state: unknown2018-01-17 11:19:13.152 +0100 Getting authorization info for user admin failed.
2018-01-17 11:19:13.161 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:19:19.289 +0100 client authd reported op command was SUCCESSFUL
2018-01-17 11:19:31.380 +0100 client dagger reported op command was SUCCESSFUL
2018-01-17 11:19:31.395 +0100 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5444): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2018-01-17 11:19:33.317 +0100
##### Non-BATCH report found (custom-dynamic-report)
2018-01-17 11:19:33.317 +0100 report generation started for 'custom-dynamic-report'
2018-01-17 11:19:33.318 +0100 ** generating report for time from 1516180773 to 1516184372

[kiwi]

Hi @soporteseguridad,

What is the PAN-OS version ?

fork() failed! errno=12 (Cannot allocate memory)

That might indicate a memory leak issue.

From the PAN-OS 8.0.6 release notes :

PAN-86353 - Fixed an issue on the Panorama management server where combinations of reports and log queries intermittently produced a slow memory leak that causes memory‐related errors such as commit failures.

Hope this helps !

Cheers !

-Kiwi.

[soporteseguridad]

PAN-OS is 8.0.3. But the commit failed occured in FW, not only running this commit from panorama. 

[kiwi · Accepted solution]

Hi @soporteseguridad,

In that case you might be hitting this :

From the PAN-OS 8.0.6 release notes : 

PAN-81100 - Fixed an issue on the firewall and Panorama management server where a memory leak caused several operations to fail, such as commits, FQDN refreshes, and content updates

Eitherway, based on the logs it looks like a memory issue so you might want to check the memory usage on the device and look for a process that might be the culprit ... a restart of that process might be a workaround for you in the meantime.

To check resource usage you can use the following command :

> show system resources follow

Cheers !

-Kiwi,