topic Re: Possible solution to slow commit in General Topics

Possible solution to slow commit

ssancho — Thu, 10 May 2012 16:01:33 GMT

Hi, regarding of the desperately slow commits in PA specially with a large number of rules and object. From our experiencie in other systems the rule shadow check is a very high CPU feature. It's sure that PA do a rule shadow and this it's in concordance with the fact that as much rules and objects you have more slow is the commit (more combinations to check)

So, could it be possible to have a check before commit to have the option of no use rule shadow?. Imagine that you are change only the name of object, probably you do not need check the shadows and I suppose the commit will be faster.

Anyone from PA could have the answer??

Thank you in advance.

Samuel

Re: Possible solution to slow commit

essnet — Thu, 10 May 2012 16:07:58 GMT

I don't think it comes from there, with same rules and during idle times I have the following:

PA-500 52 rules , 12 minutes commit
PA-200 68 rules , 1 minute commit
PA-5050 285 rules, 45 seconds commit

Re: Possible solution to slow commit

KP — Thu, 10 May 2012 18:58:39 GMT

Funny how a PA-500 is so much slower than a PA-200 on commit.

I have exactly the same experience. As far as I can tell the PA-200 does have an SSD aka compactflash with very limited (16GB) capacity, but it's way faster than the PA-500 on commits (due to this ?)

(edit) Also noticed that a PA-200 has much larger "management" memory available (2,6 GB instead of 1 GB for PA-500), could be another reason for the better performance.

Re: Possible solution to slow commit

essnet — Thu, 10 May 2012 19:35:53 GMT

I am investigating this too : my PA-500 swaps by 400MB, up to 800MB. A linux admin would be scared by this.

Try "show system resources" to have a look at your swap.

Note : PA-2020 has 1GB of RAM too, I am going to install one next week and see if/how it's different.

Re: Possible solution to slow commit

KP — Thu, 10 May 2012 19:55:39 GMT

One additional info : If I'm interpreting right, what you see with 'show system resource' is that amount of memory assigned to the control-plane. The actual firewall has more memory, but what you see is the amount left after assigning some to the data-plane. Might be wrong about this, strictly reverse-engineering from my side.

Re: Possible solution to slow commit

essnet — Thu, 10 May 2012 19:58:10 GMT

Yes I think you are assuming right, note that Dataplane doesn't suffer any slowness, only Controlplane.

Re: Possible solution to slow commit

mikand — Thu, 10 May 2012 20:18:23 GMT

What are the odds that PAN would support an upgrade of the RAM available in the units (thinking mainly of PA-500 and 2000 series who struggles with huge commit times)?

Specially when RAM memory is really cheap nowadays...

Re: Possible solution to slow commit

essnet — Thu, 10 May 2012 20:19:53 GMT

you got data about PA-20XX commit times? I am wondering if it's slow like 500 or not.

PA-5050 has 3GB of RAM, I just checked on mine

Re: Possible solution to slow commit

Jamiefitzgerald — Thu, 10 May 2012 20:58:05 GMT

PA-20XX commit times are often on par with the other older platforms with less RAM and slower processors than the newer platforms (50XX and 200). All of our newer platforms have followed all of the statements mentioned about the cost of memory and faster hardware and therefore, you will often see improvement in commit times. With all of that said, commit times are very much determined by what changes are being made, by the amount of configuration you are commiting, the amount of logging, and the features running on the FW.

Re: Possible solution to slow commit

Jamiefitzgerald — Thu, 10 May 2012 21:01:35 GMT

As a note to essnet, PAN-OS will not recognize added RAM to the underlying OS.

Re: Possible solution to slow commit

ssancho — Thu, 10 May 2012 21:13:49 GMT

Two things:

-I can confirm to you that PA 2020 and PA2050 has the same commit times than PA-500. This week about 8 or 9 minutes in PA-2050 for each commit.

-thanks jfitz-gerald for your help but what about the origin of this topic?, could it be posible to disable shadow rules check, could it be one of the reasons for this commit times?. We know that the time depends on the size of configuration, logging, number of objects and number of rules, but often this configurations are necesary and the perception of client is a very slow system.

Thanks

Samuel

Re: Possible solution to slow commit

dkhoe — Thu, 10 May 2012 22:59:13 GMT

Hah... our 4050s sometimes take up to 20 minutes to commit...

Re: Possible solution to slow commit

essnet — Thu, 10 May 2012 23:01:10 GMT

jfitz-gerald I hope to get in touch with you very soon about this topic over phone with France PA representatives.

This place is a geek nest and I love it, I see that I am not alone and isolated about this problem despite what I was told and others also understood that RAM is probably the problem may be among other things.

Note this is extremly problematic in production envrionnement : you get alerted of a problem, you login on FW (takes 2 minutes), you look at logs, make a change ( 10 minutes lost), make new tests, still doesn't work, new commit (10 more minutes). This is extremly true when you deploy a new box, like as a replacement to another vendor : you know that many rules will fail after migration, you have 1 or 2 hours deadline to make the move. 1 hour is just 3 or 4 commits max in my case.

10 minutes is a hell of an eternity in production, in addition I don't get faster commits during idle or busy hours.

I am sure you will provide us a solution, but I hope it won't take too long because I already had problems with top management who I had to explain what takes us so long to fix problems.

Sell us PA-505, 2022 and 2055 models !

Re: Possible solution to slow commit

NGS_SOC — Thu, 10 May 2012 23:32:43 GMT

20XX series with faster disks , I tried WD Black Caviar 64 MB cache, help reducing commit times. Using a PA-2020 4.1.6 in production I've seen a mean reduction up to 1-2 mins.

Also rebooting the machine, better if in HA environment, can help specially when for some reasons commit take 10+ minutes.

A suggestion for PAN: please upgrade hard disk and provide 2000 & 500 series with SSD, 200 & 5000 got it, why not the other series?

Re: Possible solution to slow commit

mikand — Fri, 11 May 2012 06:39:07 GMT

I think you see an improvement due to the original fault where way to much swap is being used (reports in this forum of 400-800 MB of swap during commit).

When swap is being used the box will use the harddrive as "memory" and of course if you use an SSD instead the commit time will lower.

But the proper fix is to not use swap at all - meaning increase amount of ram in the box so it wont swap at the first place (unless there is some software fix so the box wont use this much memory during commit).

Perhaps PA could address this with an upgrade kit which wont void the warranty?

The hardware needed (just an example to see the prices):

KINGSTON 8GB 1333MHZ DDR3 CL9 HYPERX BLU (2X4GB)
~60 dollar + VAT

SAMSUNG 830 SERIES 256GB SSD SATA/600 MLC
~330 dollar + VAT

(there is a 512GB model of above SSD for roughly 640 dollar + VAT)

So an upgrade for less than 500 dollar + VAT (including work) should be possible (or less than 800 dollar if we go for the 512GB model instead which I think should be preferred).