Auto-commit failing: interfaces down, not able to force commit

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Auto-commit failing: interfaces down, not able to force commit

L3 Networker

We are struggling with the following error and Palo Alto TAC is not able to provide the proper support, they are just asking us to do an RMA or to factory reset, but the truth is that we are having the same issue in 2 different firewall clusters with different configs and specs.

 

After the firewalls powers on/reboot the "auto-commit" gets stuck at 55%:

 

MarcelST_1-1606948327447.png

 

Once it get canceled or it fails, we try to run a commit and we get the following error:

 

MarcelST_0-1606948079053.png

 

On CLI  with: "tail follow yes mp-log devsrv.log" we get:

 

2020-12-02 23:38:04.925 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:09.927 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:09.930 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:09.930 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:14.931 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:14.935 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:14.935 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push
2020-12-02 23:38:19.934 +0100 debug: pan_ctrl_need_cfg_cb(pan_controller_proc.c:1359): sysd_notify_change: get need config notification from cfgpush.s1.comm.need-cfg
2020-12-02 23:38:19.938 +0100 debug: pan_cfgagent_can_receive_cfg(pan_cfgagent.c:239): cfgagent's previous config still in use
2020-12-02 23:38:19.938 +0100 debug: pan_log_handle_needcfg(pan_log_handler.c:396): ctrl can't receive config push

 

 

We have gone through all the solutions I could find out there but nothing worked and Palo Alto support seems to not be able to help.

1 accepted solution

Accepted Solutions

We were finally able to identify the issue with the support of the Palo Alto engineer assigned to our account.

 

It's a bug with EDL that starts at PAN-os v9.0.0. All our firewalls that where at that version or a newer one where facing the issue, while the firewalls on lower versions where not. Furthermore, if you downgrade them it gets solved.

 

It seems related to some characters present on EDL text file (%, $, *,.,...) that makes the auto-commit and the EDL refresh process fail starting on PANOS v9.0.0. It has not yet being identified as a bug though, but hopefully it will soon.

View solution in original post

9 REPLIES 9

Cyber Elite
Cyber Elite

Have you tried, from CLI, do to a "commit force"?

 

if you are getting the same/similar error, then loading a config from maintenance mode is what I would recommend.

I do not think RMA is needed, but a factory-reset may work.

Do you have an exported/saved copy of the configuration?

If the FW are in clusters (thinking HA) can you have the 2ndary/back back be primary (if its AutoCommit is successful) when you tshoot the primary FW?

 

You have choices.  Which way do you want to go?

Help the community: Like helpful comments and mark solutions

Thanks Steve. "commit force" did not helped. 

 

We just tried going into maintenance mode and reverted to a previous software version, that allowed the "auto-commit" to happen, but right after that the underlying error is still there: can't commit from Panorama or the firewall itself (stuck at some random %), can't install content updates, can't update/ugprade,...

We were finally able to identify the issue with the support of the Palo Alto engineer assigned to our account.

 

It's a bug with EDL that starts at PAN-os v9.0.0. All our firewalls that where at that version or a newer one where facing the issue, while the firewalls on lower versions where not. Furthermore, if you downgrade them it gets solved.

 

It seems related to some characters present on EDL text file (%, $, *,.,...) that makes the auto-commit and the EDL refresh process fail starting on PANOS v9.0.0. It has not yet being identified as a bug though, but hopefully it will soon.

What was your ticket number?   I believe we are running into the same issue.

Did you ever resolve this? What was your path to get past this bug?

 

We are trying to upgrade multiple 220s from 9.1.13 to 10.0.X and it's failing on the auto commit

Search for any of the unsupported characters listed above, and remove those from the EDL/whatever is hosting that first.

yeah, the only EDLs I have configured are the predetermined JUNOS ones, "high risk IP", "malicious IP", etc. I'm also blocking by countries which I believe is an EDL as well. No custom EDLs with weird characters in them. Just an error message that says

 

Error: Profile compiler : invalid profile name default
Error: Profile compiler : Global section error
Error: Profile compiler : parsing config error
(Module: device)
Commit failed
Failed to commit policy to device

Are you sure it's the EDL that is causing your failure?

In this case, no. It doesn't appear the EDL was causing it. We removed all the EDLs and still experienced the issue. In case anyone else is experiencing the same issue we are having, here is our resolution. We were able to resolve our issue by getting a PA engineer (after escalating with 2 others) with root access to the box and delete the .global-fin file under /opt/pancfg/mgmt/global/ from root shell followed by a management server restart. Still no word on why that was necessary. One of the engineers told us that upgrading our FWs through Panorama is not recommended and that they should only be upgraded through the FW UI itself. We aren't sure why that is being suggested since PA specifically refers to upgrading via Panorama in their documentation and are seeking more information on the case.

  • 1 accepted solution
  • 22858 Views
  • 9 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!