Auto-commit failing: interfaces down, not able to force commit

[MarcelST]

We are struggling with the following error and Palo Alto TAC is not able to provide the proper support, they are just asking us to do an RMA or to factory reset, but the truth is that we are having the same issue in 2 different firewall clusters with different configs and specs.

After the firewalls powers on/reboot the "auto-commit" gets stuck at 55%:

Once it get canceled or it fails, we try to run a commit and we get the following error:

On CLI  with: "tail follow yes mp-log devsrv.log" we get:

2020-12-02 23:38:04.925 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:09.927 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:09.930 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:09.930 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:14.931 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:14.935 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:14.935 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:19.934 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:19.938 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:19.938 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push

We have gone through all the solutions I could find out there but nothing worked and Palo Alto support seems to not be able to help.

[MarcelST · Accepted solution]

We were finally able to identify the issue with the support of the Palo Alto engineer assigned to our account.

It's a bug with EDL that starts at PAN-os v9.0.0. All our firewalls that where at that version or a newer one where facing the issue, while the firewalls on lower versions where not. Furthermore, if you downgrade them it gets solved.

It seems related to some characters present on EDL text file (%, $, *,.,...) that makes the auto-commit and the EDL refresh process fail starting on PANOS v9.0.0. It has not yet being identified as a bug though, but hopefully it will soon.

[SteveCantwell]

Have you tried, from CLI, do to a "commit force"?

if you are getting the same/similar error, then loading a config from maintenance mode is what I would recommend.

I do not think RMA is needed, but a factory-reset may work.

Do you have an exported/saved copy of the configuration?

If the FW are in clusters (thinking HA) can you have the 2ndary/back back be primary (if its AutoCommit is successful) when you tshoot the primary FW?

You have choices.  Which way do you want to go?

[MarcelST]

Thanks Steve. "commit force" did not helped. 

We just tried going into maintenance mode and reverted to a previous software version, that allowed the "auto-commit" to happen, but right after that the underlying error is still there: can't commit from Panorama or the firewall itself (stuck at some random %), can't install content updates, can't update/ugprade,...

[MarcelST · Accepted solution]

We were finally able to identify the issue with the support of the Palo Alto engineer assigned to our account.

It's a bug with EDL that starts at PAN-os v9.0.0. All our firewalls that where at that version or a newer one where facing the issue, while the firewalls on lower versions where not. Furthermore, if you downgrade them it gets solved.

It seems related to some characters present on EDL text file (%, $, *,.,...) that makes the auto-commit and the EDL refresh process fail starting on PANOS v9.0.0. It has not yet being identified as a bug though, but hopefully it will soon.