Commit Fail Phase1 sslvpn

[minow]

hey

i am gtting commit fail on phase1 abort, and it looks like the SSL VPN proccess how can i troubleshoot it ? 

show management-clients

Client PRI State Progress
-------------------------------------------------------------------------
routed 30 P1-abort 0
ha_agent 25 P1-abort 0
device 20 P1-abort 0
ikemgr 10 P1-abort 0
keymgr 10 init 0 (op cmds only)
logrcvr 10 P1-abort 0
dhcpd 10 P1-abort 0
varrcvr 10 P1-abort 0
l3svc 10 P1-abort 0
sslvpn 10 P1-abort 0 *
rasmgr 10 P1-abort 0
useridd 10 P1-abort 0
satd 10 P1-abort 0
websrvr 10 P1-abort 0
sslmgr 10 P1-abort 0
authd 10 P1-abort 0
pppoed 10 P1-abort 0
dnsproxyd 10 P1-abort 0
cryptod 10 P1-abort 0
dagger 10 init 0 (op cmds only)
l2ctrld 10 P1-abort 0

Overall status: P1-abort. Progress: 0
Warnings:
Errors:

[nextgenhappines]

can you show job id xx and include the exact message?

Anything interesting on the ms.log or devsrv.log?

use less mp-log ms.log ir less mp-log devsrv.log and go back around the time when the commit was going on..

[minow]

show jobs id 4706

Enqueued ID Type Status Result Completed
--------------------------------------------------------------------------
2017/02/16 09:22:06 4706 Commit FIN FAIL 09:23:15
Warnings:Duplicate certificate subject found:
/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
/OU=Domain Control Validated/CN=*.mydomain.com

Details:Commit failed

mplog los

2017-02-16 09:23:08.597 -0600 client satd reported Phase 1 was SUCCESSFUL
2017-02-16 09:23:09.665 -0600 client l2ctrld reported Phase 1 was SUCCESSFUL
2017-02-16 09:23:10.773 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client l2ctrld went down from 99 to 98
2017-02-16 09:23:14.391 -0600 Error: pan_mgmt_client_table_do_commit(pan_cfg_commit_jobs.c:3575): phase 1 failed
2017-02-16 09:23:14.396 -0600 client routed reported error: config commit phase 1 aborted(Module: routed)
2017-02-16 09:23:14.396 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.402 -0600 client device reported error: Config commit phase 1 aborted(Module: device)
2017-02-16 09:23:14.402 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.406 -0600 client ikemgr reported error: panike_daemon phase 1 aborted(Module: ikemgr)
2017-02-16 09:23:14.406 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.412 -0600 client dhcpd reported error: config commit phase 1 aborted(Module: dhcpd)
2017-02-16 09:23:14.412 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.415 -0600 client varrcvr reported error: config commit phase 1 aborted(Module: varrcvr)
2017-02-16 09:23:14.415 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.419 -0600 client l3svc reported error: modhttpd phase 1 aborted(Module: l3svc)
2017-02-16 09:23:14.419 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.422 -0600 client sslvpn reported error: modsslvpn phase 1 aborted(Module: sslvpn)
2017-02-16 09:23:14.422 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.426 -0600 client rasmgr reported error: rasmgr phase 1 aborted(Module: rasmgr)
2017-02-16 09:23:14.426 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.433 -0600 client satd reported error: satd phase 1 aborted(Module: satd)
2017-02-16 09:23:14.433 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.438 -0600 client websrvr reported error: modappweb phase 1 aborted(Module: websrvr)
2017-02-16 09:23:14.438 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.454 -0600 client pppoed reported error: config commit phase 1 aborted(Module: pppoed)
2017-02-16 09:23:14.454 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.457 -0600 client dnsproxyd reported error: config commit phase 1 aborted(Module: dnsproxyd)
2017-02-16 09:23:14.457 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.461 -0600 client cryptod reported error: config commit phase 1 aborted(Module: cryptod)
2017-02-16 09:23:14.461 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.462 -0600 EBL cfg(0x118799a0, 0) Reverting EBLs
2017-02-16 09:23:14.467 -0600 Error: pan_cfg_commit_to_local_device(pan_cfg_commit_handler.c:2042): Commit failed
2017-02-16 09:23:14.475 -0600 client l2ctrld reported error: config commit phase 1 aborted(Module: l2ctrld)
2017-02-16 09:23:14.475 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:14.804 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client routed went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client ha_agent went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client device went down from 5 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client ikemgr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client logrcvr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client dhcpd went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client varrcvr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client l3svc went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client rasmgr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client useridd went down from 70 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client satd went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client websrvr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client sslmgr went down from 98 to 0
2017-02-16 09:23:14.805 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client authd went down from 98 to 0
2017-02-16 09:23:14.806 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client pppoed went down from 98 to 0
2017-02-16 09:23:14.806 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client dnsproxyd went down from 98 to 0
2017-02-16 09:23:14.806 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client cryptod went down from 98 to 0
2017-02-16 09:23:14.806 -0600 Error: pan_mgmt_client_table_get_current_progress(pan_cfg_commit_jobs.c:3973): commit progress for client l2ctrld went down from 98 to 0
2017-02-16 09:23:15.396 -0600 client useridd reported Phase 1 FAILED
2017-02-16 09:23:15.396 -0600 Error: pan_mgmt_client_p1done_callback(pan_cfg_commit_jobs.c:258): but there was no outstanding Phase 1. Ignoring
2017-02-16 09:23:15.403 -0600 client useridd reported error: Config commit phase 1 aborted(Module: useridd)
2017-02-16 09:23:15.403 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:17.380 -0600 Could not find url vendor, returning paloaltonetworks as default
2017-02-16 09:23:25.504 -0600 UNKNOWN TID: MODIFY 1954 [RSP,v1] (100/27) { 'flgs': 0x2, 'pfx': [ sw.cfgagent.device., ], 'spec': abortp1, 'ss': [ ], } obj: [ True, ]
2017-02-16 09:23:27.307 -0600 client device reported error: Error: Internal Error(Module: device)
2017-02-16 09:23:27.307 -0600 Error: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
2017-02-16 09:23:27.309 -0600 client device reported Phase 1 FAILED
2017-02-16 09:23:27.309 -0600 Error: pan_mgmt_client_p1done_callback(pan_cfg_commit_jobs.c:258): but there was no outstanding Phase 1. Ignoring
2017-02-16 09:23:27.675 -0600 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:367): failed to fetch cfg.gpdatafile-release-date

debicelog logs

2017-02-16 09:22:59.094 -0600 [TDB] Loading tdb cache /opt/pancfg/mgmt/content//cache/70000//tdb.cache.ser-0 with wildfire 0/0 virus 0/0
2017-02-16 09:22:59.094 -0600 calc md5
2017-02-16 09:23:14.401 -0600 Config commit phase1 abort
2017-02-16 09:23:14.401 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:14.405 -0600 Error: cfgagent_modify_callback(pan_cfgagent.c:83): Modify string (sw.mgmt.runtime.clients.device.err) error: USER (1)
2017-02-16 09:23:15.412 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:16.421 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:17.432 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:18.433 -0600 End of parsing custom threat
2017-02-16 09:23:18.441 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:19.451 -0600 tdb compile flag is still up, abort thread wait 1 second
2017-02-16 09:23:19.499 -0600 [Cache] Load /opt/pancfg/mgmt/content//cache/70000//tdb.cache.ser-0 success
2017-02-16 09:23:19.515 -0600 [TDB] stats virus pattern 813145 , 104826
2017-02-16 09:23:19.519 -0600 Warning: pan_ctrl_compile_tdb(pan_config_handler_sysd.c:487): config commit aborted
2017-02-16 09:23:19.519 -0600 TDB compilation done, return 0
2017-02-16 09:23:20.461 -0600 kill SIGUSR1 to pid 0
2017-02-16 09:23:24.812 -0600 Error: pan_com_sysd_send_msg(pan_com_sysd.c:239): sysd_modify_obj() failed: TIMEOUT
2017-02-16 09:23:26.749 -0600 Error: pan_com_send_msg_through_sysd(pan_com_mp.c:337): pan_com_reset_sysd_reply_msg() failed
2017-02-16 09:23:26.749 -0600 Error: pan_status_send_msg(pan_status_handler.c:1208): pan_com_send_msg failed
2017-02-16 09:23:26.749 -0600 Warning: pan_ctrl_save_config(pan_config_handler_sysd.c:1686): config commit aborted
2017-02-16 09:23:26.749 -0600 Error: pan_ctrl_compile_cfg(pan_config_handler_sysd.c:1891): pan_ctrl_save_config() failed
2017-02-16 09:23:26.749 -0600 Error: pan_config_handler_sysd(pan_config_handler_sysd.c:2174): pan_ctrl_compile_cfg() failed
2017-02-16 09:23:26.749 -0600 Error: pan_ctrl_parse_config(pan_controller_proc.c:375): pan_config_handler_sysd() failed
2017-02-16 09:23:25.503 -0600 UNKNOWN TID: MODIFY 1268 [RSP,v1,URG] (100/26) { 'flgs': 0x1022, 'pfx': [ sw.comm.s1.dp0., ], 'spec': runtime.com, 'ss': [ ], } obj: [ None, ]
2017-02-16 09:23:27.215 -0600 Error: pan_ctrl_config_phase1(pan_controller_proc.c:725): pan_ctrl_parse_config() failed
2017-02-16 09:23:27.215 -0600 Config commit phase1 failed
2017-02-16 09:23:27.308 -0600 Error: cfgagent_modify_callback(pan_cfgagent.c:83): Modify string (sw.mgmt.runtime.clients.device.err) error: USER (1)
2017-02-16 09:23:27.309 -0600 Error: bool_modify_callback(pan_cfgagent.c:101): Modify boolean (sw.mgmt.runtime.clients.device.p1done) error USER (1)

[minow]

the wiered issue i cannot login to the webUI , show certificate warning but i get blank page

[nextgenhappines]

try restart the web-server and see if you can login in via webUI again.

debug software restart process web-server

Can you roll back the change that you just made and review what is changed?

[minow]

tried restarting

device server

web-server

tried to roll back to configuration from the same day morning

nothing helped.

this is a remote site PA so i am not comfertable in restarting the device right now

[minow]

the problem was with diskspace on the PA .

figure it out after i tried to view local configuration and it failed.

after clearing some core logs and old system logs i was able to commit locally and from panorama

[minow]

does anyone have any idea on how to debug those kind of issue ?

support ticket might not be availible as fast as knowing this when it is needed.

[nextgenhappines]

I usually write down the job fail message, tail the ms.log and devsrv.log while commit is running, check show management client status

1.  try google

2. search on the community site.  

3  some time you may need to restart some of the process, before you restart it, take multiple backtraces or core that process if possible , so you have something for TAC to review if in fact by restart that process fixed the problem. 

if I am still stuck, I just call TAC for help and docuement the commands that the TAC engineer used and ask TAC engineer questions, they are pretty cool to provide explaination, you will pick up tricks and understanding in no time.     

If you are able to fix the problem, I tried to open a case and included the tech support file and the backtrace log and core file for TAC to review it.  It could be something new that they have not seen before.  

[kiwi]

Hi @minow,

A good starting point on how to start debugging commit errors might be the following blog :

https://live.paloaltonetworks.com/t5/Community-Blog/Why-is-this-commit-not-working/ba-p/140491

It explains where to look and also how to follow the logs during commits.

If it's still unclear on why the commit is failing, at least you can gather all the data for TAC.

Good luck !

-Kim.

[wtian]

Access Denied for the debugging commit link. 

[BPry]

@wtian,

For commit failures you can follow the below KB. 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PMb2CAG