topic Re: Problem with Panorama commit in General Topics

Problem with Panorama commit

Bittereinder — Thu, 21 Mar 2019 15:22:13 GMT

Hello there!

I have a problem when I try to push the commit to Panorama. This commit is added to the queue, but then nothing else happens. The odd thing is that nothing is queued in the firewall queue...

Thanks in advance.

Re: Problem with Panorama commit

raphaelc — Thu, 21 Mar 2019 20:30:20 GMT

What software version are you using?

Re: Problem with Panorama commit

Bittereinder — Thu, 21 Mar 2019 22:46:17 GMT

It's Panorama version 8.1.6.

Re: Problem with Panorama commit

jeremy.larsen — Fri, 22 Mar 2019 15:12:25 GMT

So does it FAIL eventually?

Re: Problem with Panorama commit

Bittereinder — Sat, 23 Mar 2019 12:37:34 GMT

Yes, jeremy.larsen, it fails. No error is shown, but the commit is not done. This task appears to be on the queue, but is never done.

Thanks and regards.

Re: Problem with Panorama commit

jeremy.larsen — Mon, 25 Mar 2019 22:48:09 GMT

I'm confused. If it is still in the queue, then that sounds like it is hung but never actually fails? By fail, I mean you get a commit failure in the logs with details about the failure.

Re: Problem with Panorama commit

ebonjour — Wed, 27 Mar 2019 20:51:51 GMT

I have the same problem! I'm running 8.0.15 and I've had this issue since upgrading to 8.0 from 7.1.

I only have this issue on firewalls with HA. I can get the commits to go through after restarting panorama.

My TAC case is about 3 months old at this point.

Edit: I never see the commit hit the firewalls, it stays in queue on the panorama.

Re: Problem with Panorama commit

Bittereinder — Wed, 27 Mar 2019 21:10:30 GMT

You're right, jeremy.larsen, I did not express it right. The commit does not fail, it was simply never made.

But good news: this problem was solved after rebooting the device. 🙂

Re: Problem with Panorama commit

Bittereinder — Wed, 27 Mar 2019 21:14:17 GMT

@ebonjour I was to quick answering you without reading that you have already tried rebooting the device. It worked at least for me... Regards.

Re: Problem with Panorama commit

jeremy.larsen — Thu, 28 Mar 2019 13:14:21 GMT

I'd be curious to see if you pulled a snapshot and reinstalled Panorama from scratch on the same version if this continued to happen. Perhaps some residual garbage from the upgrade process? I know that sounds painful.

Re: Problem with Panorama commit

ebonjour — Thu, 28 Mar 2019 13:46:14 GMT

@Bittereinder wrote:
@ebonjour I was to quick answering you without reading that you have already tried rebooting the device. It worked at least for me... Regards.

If you are having the same problem I am, then the commit queueing will occuring again. Usually this takes about 5-10 minutes, but sometimes longer.

Re: Problem with Panorama commit

ebonjour — Thu, 28 Mar 2019 13:49:31 GMT

@jeremy.larsen wrote:
I'd be curious to see if you pulled a snapshot and reinstalled Panorama from scratch on the same version if this continued to happen. Perhaps some residual garbage from the upgrade process? I know that sounds painful.

After 3 months of this I would try anything, but this does sound really painful. The worst part of this is that diffs fail, these actually timeout after 4 minutes and 30 seconds, unlike commits that do not have a failure timer (kinda scary!).

Re: Problem with Panorama commit

ebonjour — Thu, 11 Apr 2019 20:50:38 GMT

After 4 months I asked for esclation and after a couple days received a request to make a simple change.

Panorama -> Setup -> Logging and Reporting Settings -> Buffered Log Fowarding from Device - enable it.

It took 3 hours to impact one set of HA firewalls and 6 hours another another set. Not enabling this requires the firewalls ACK every single log sent to Panorama which increases mgmt CPU usage and causes issues like I had.

Not sure if this setting is default in later releases, it should be. I've had this panorama install since 5.1 and upgraded since then, issue starting happening after upgrading to 8.0.x.

Re: Problem with Panorama commit

jeremy.larsen — Fri, 12 Apr 2019 13:24:52 GMT

I can confirm this is the default on newer installs. Here is a screenshot from a new install from a few weeks ago in the lab. Glad you figured this out!