Problem with Panorama commit

[Bittereinder]

Hello there!


I have a problem when I try to push the commit to Panorama. This commit is added to the queue, but then nothing else happens. The odd thing is that nothing is queued in the firewall queue...


Thanks in advance.

[raphaelc]

What software  version are you using?  

[Bittereinder]

It's Panorama version 8.1.6.

[jeremy.larsen]

So does it FAIL eventually?

[Bittereinder]

Yes, jeremy.larsen, it fails. No error is shown, but the commit is not done. This task appears to be on the queue, but is never done.

Thanks and regards.

[jeremy.larsen]

I'm confused.  If it is still in the queue, then that sounds like it is hung but never actually fails?  By fail, I mean you get a commit failure in the logs with details about the failure.

[ebonjour]

I have the same problem!  I'm running 8.0.15 and I've had this issue since upgrading to 8.0 from 7.1.

I only have this issue on firewalls with HA.  I can get the commits to go through after restarting panorama.

My TAC case is about 3 months old at this point.

Edit:  I never see the commit hit the firewalls, it stays in queue on the panorama.

[Bittereinder]

You're right, jeremy.larsen, I did not express it right. The commit does not fail, it was simply never made.

But good news: this problem was solved after rebooting the device. 🙂

[Bittereinder]

@ebonjour I was to quick answering you without reading that you have already tried rebooting the device. It worked at least for me... Regards.

[jeremy.larsen]

I'd be curious to see if you pulled a snapshot and reinstalled Panorama from scratch on the same version if this continued to happen.  Perhaps some residual garbage from the upgrade process?  I know that sounds painful.

[ebonjour]

---

@Bittereinder wrote:

@ebonjour I was to quick answering you without reading that you have already tried rebooting the device. It worked at least for me... Regards.

---

If you are having the same problem I am, then the commit queueing will occuring again.  Usually this takes about 5-10 minutes, but sometimes longer.

[ebonjour]

---

@jeremy.larsen wrote:

I'd be curious to see if you pulled a snapshot and reinstalled Panorama from scratch on the same version if this continued to happen.  Perhaps some residual garbage from the upgrade process?  I know that sounds painful.

---

After 3 months of this I would try anything, but this does sound really painful.  The worst part of this is that diffs fail, these actually timeout after 4 minutes and 30 seconds, unlike commits that do not have a failure timer (kinda scary!).

[ebonjour]

After 4 months I asked for esclation and after a couple days received a request to make a simple change.

Panorama -> Setup -> Logging and Reporting Settings -> Buffered Log Fowarding from Device - enable it.

It took 3 hours to impact one set of HA firewalls and 6 hours another another set.  Not enabling this requires the firewalls ACK every single log sent to Panorama which increases mgmt CPU usage and causes issues like I had. 

Not sure if this setting is default in later releases, it should be.  I've had this panorama install since 5.1 and upgraded since then, issue starting happening after upgrading to 8.0.x.

[jeremy.larsen]

I can confirm this is the default on newer installs.  Here is a screenshot from a new install from a few weeks ago in the lab.  Glad you figured this out!