Auto-commit failing: interfaces down, not able to force commit

[MarcelST]

We are struggling with the following error and Palo Alto TAC is not able to provide the proper support, they are just asking us to do an RMA or to factory reset, but the truth is that we are having the same issue in 2 different firewall clusters with different configs and specs.

After the firewalls powers on/reboot the "auto-commit" gets stuck at 55%:

Once it get canceled or it fails, we try to run a commit and we get the following error:

On CLI  with: "tail follow yes mp-log devsrv.log" we get:

2020-12-02 23:38:04.925 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:09.927 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:09.930 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:09.930 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:14.931 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:14.935 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:14.935 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:19.934 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:19.938 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:19.938 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push

We have gone through all the solutions I could find out there but nothing worked and Palo Alto support seems to not be able to help.

[S.Cantwell]

Have you tried, from CLI, do to a "commit force"?

if you are getting the same/similar error, then loading a config from maintenance mode is what I would recommend.

I do not think RMA is needed, but a factory-reset may work.

Do you have an exported/saved copy of the configuration?

If the FW are in clusters (thinking HA) can you have the 2ndary/back back be primary (if its AutoCommit is successful) when you tshoot the primary FW?

You have choices.  Which way do you want to go?

[MarcelST]

Thanks Steve. "commit force" did not helped. 

We just tried going into maintenance mode and reverted to a previous software version, that allowed the "auto-commit" to happen, but right after that the underlying error is still there: can't commit from Panorama or the firewall itself (stuck at some random %), can't install content updates, can't update/ugprade,...

[MarcelST · Accepted solution]

We were finally able to identify the issue with the support of the Palo Alto engineer assigned to our account.

It's a bug with EDL that starts at PAN-os v9.0.0. All our firewalls that where at that version or a newer one where facing the issue, while the firewalls on lower versions where not. Furthermore, if you downgrade them it gets solved.

It seems related to some characters present on EDL text file (%, $, *,.,...) that makes the auto-commit and the EDL refresh process fail starting on PANOS v9.0.0. It has not yet being identified as a bug though, but hopefully it will soon.

[Sec101]

What was your ticket number?   I believe we are running into the same issue.

[JoshuaSanders]

Did you ever resolve this? What was your path to get past this bug?

We are trying to upgrade multiple 220s from 9.1.13 to 10.0.X and it's failing on the auto commit

[Sec101]

Search for any of the unsupported characters listed above, and remove those from the EDL/whatever is hosting that first.

[JoshuaSanders]

yeah, the only EDLs I have configured are the predetermined JUNOS ones, "high risk IP", "malicious IP", etc. I'm also blocking by countries which I believe is an EDL as well. No custom EDLs with weird characters in them. Just an error message that says

Error: Profile compiler : invalid profile name default
Error: Profile compiler : Global section error
Error: Profile compiler : parsing config error
(Module: device)
Commit failed
Failed to commit policy to device

[Sec101]

Are you sure it's the EDL that is causing your failure?

[JoshuaSanders]

In this case, no. It doesn't appear the EDL was causing it. We removed all the EDLs and still experienced the issue. In case anyone else is experiencing the same issue we are having, here is our resolution. We were able to resolve our issue by getting a PA engineer (after escalating with 2 others) with root access to the box and delete the .global-fin file under /opt/pancfg/mgmt/global/ from root shell followed by a management server restart. Still no word on why that was necessary. One of the engineers told us that upgrading our FWs through Panorama is not recommended and that they should only be upgraded through the FW UI itself. We aren't sure why that is being suggested since PA specifically refers to upgrading via Panorama in their documentation and are seeking more information on the case.