How to debug commit?

[mhuels]

Hi folks,

does anybody know how to debug the failing commits on a Palo Alto Firewall? The onliest what i can see is "failure on pushing config to device".

user@pan> show jobs all

Enqueued                     ID             Type    Status Result Completed
--------------------------------------------------------------------------

2012/04/27 15:12:46           7           Commit       FIN   FAIL 15:14:38

2012/04/27 14:56:02           6          AutoCom       FIN   FAIL 14:57:43

2012/04/27 14:54:17           5          AutoCom       FIN   FAIL 14:55:59

2012/04/27 14:52:27           4          AutoCom       FIN   FAIL 14:54:10

2012/04/27 14:49:51           3          AutoCom       FIN   FAIL 14:52:19

2012/04/27 14:47:10           2          AutoCom       FIN   FAIL 14:49:46

2012/04/27 14:44:19           1          AutoCom       FIN   FAIL 14:47:05

mfg

Manfred

[rmonvon]

Hi...Please try command 'show jobs id 7' to view the details of the commit job.  It appears you're getting a FAILure on the AutoCom job.  You may want to try 'commit force' to override.

Thanks.

[sspringer · Accepted solution]

You can also run 'show management-clients' which will show the client process failing.

Normally there are error messages inside the ms.log or devsrv.log in management-plane logs. The commands below will view the last 100 lines of the files. These files can also be viewed with 'less mp-log ms.log'

> tail lines 100 mp-log ms.log

> tail lines 100 mp-log devsrv.log

If the reason for the failure is not clear, I would recommend opening a case with your support team for further debugging.

- Stefan

[mhuels]

Hi rmonvon

show jobs id is no very meaningful:

user@pan> show jobs id 7

Enqueued                     ID             Type    Status Result Completed
--------------------------------------------------------------------------
2012/04/27 15:12:46           7           Commit       FIN   FAIL 15:14:38 
Warnings:
Details:device: config push error
Commit failed

I will try to debug the commit first, because i am worry about making a "commit force" and getting a totally defective firewall.

Thanks for your hints.

Manfred

[mhuels]

the first error i can see is:

Apr 30 13:02:06 Error: pan_schema_verify_enum(pan_schema_verify.c:699): 'win\id_h_internet_voll' is not an allowed keyword near line 0

i deleted all entries with 'id_h_internet_voll' from the xml-configfile. But there still remains the error 'win\id_h_internet_voll'. Seems to be a config problem outside the XML config.

regards

Manfred

[mikand]

Do you have another box to perform tests on?

Since a reboot would make it autocommit and in case it cannot commit you would end up with a (from the client/server point of view) dead unit.

I wonder if you export running-config.xml and import it (under a new name so you wont end up with two "running-config" :smileysilly:) in another box (with the same PANOS and hardware model) - do you get the same error?

[mhuels]

Hi mikand,

i make my tries on a backup machine, but i have no spare firewall to try on another hardware.

The PAN support recommended a "factory reset" and consecutively a config load of a preserved XML-File.

After the factory reset the firewall runs fine at first. By trying to load the old config, the GUIs tells "... import successfull". But the rules and objects are not present.

The ms.log says

"
May 02 11:58:24 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch
May 02 11:58:25 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:809): failed to fetch: NO_MATCHES
ls: /opt/panlogs/logdb/appstatdb/1/: No such file or directory
May 02 12:00:04 Error: pan_dir_exists(pan_fs.c:183): entry exists but it's a file
May 02 12:00:04 Warning: ha_cfg_filesync_md5sum(ha_cfg.c:1104): All values seem to be disconnected from peer, giving back error or md5sum failue
May 02 12:00:04 Error: pan_mgmt_ha_set_dsmd5sum(pan_mgmt_ha.c:170): failed to calculate disk-state md5sum
May 02 12:06:08 Error: pan_dir_exists(pan_fs.c:183): entry exists but it's a file
May 02 12:06:09 Warning: ha_cfg_filesync_md5sum(ha_cfg.c:1104): All values seem to be disconnected from peer, giving back error or md5sum failue
May 02 12:06:09 Error: pan_mgmt_ha_set_dsmd5sum(pan_mgmt_ha.c:170): failed to calculate disk-state md5sum

"

I cannot detect any hints for further troubleshooting.

Next i will try to put this firewall by hand to the firewallcluster and synchronize the config over the running firewall.

mfg

Manfred

[mhuels]

Synchronizing within the cluster fails too. HA-Sync and the manual commit fails without any usefull log entry.

Couriously the validation dialog says:

May 03 12:57:22 Configuration is valid

Palo Alto Networks has now climbed a couple of points on my personal list of the world most evil software. Its not so far away from Lotus Notes any more 😉

mfg

Manfred

[rmonvon]

Manfred...Please contact Support to get more assistance to diagnose this issue.  Thanks.

[mhuels]

Hi rmonvon,

i did so a couple of days ago. For today PAN promises a remote desktop session. On my humble opinion the troubleshooting capabilities within the firewall should be improved.

mfg

Manfred

[rmonvon]

Thank you for your patient and understanding.  Support will be able to review the system logs which has more details on the failure.