Palo Alto PA-3050 100 % CPU

[Transporter]

These were upgraded from 7.0.6 to 7.1.2 on 31 May.  Since then we are suffering from the data plane very frequently using 100% CPU.

For example:

admin@PA-3050-5(active)> show running resource-monitor hour

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:

core    0       1       2       3       4       5

     avg max avg max avg max avg max avg max avg max

       *   *   7  35  16 100  17 100  19 100  20 100

       *   *   5  45  10  73  11  74  13  75  13  75

       *   *   4  15   8  60   8  60  10  61  10  60

       *   *   4   6   7  12   8  13  10  15  10  16

       *   *   4   6   7  19   8  19   9  22  10  22

       *   *   4   6   7  15   8  15   9  19  10  19

       *   *   4   7   7  13   8  14  10  16  10  17

       *   *   4   7   8  14   8  15  10  17  11  18

       *   *  24  89  45 100  46 100  48 100  48 100

       *   *  42  81  81 100  81 100  82 100  82 100

       *   *  41  85  85 100  85 100  86 100  86 100

       *   *  27  60  69 100  70 100  71 100  72 100

       *   *  35  79  77 100  77 100  78 100  78 100

       *   *  24  77  51 100  52 100  54 100  55 100

       *   *  28  79  59 100  59 100  61 100  62 100

       *   *  30  83  65 100  66 100  68 100  68 100

       *   *  29  96  63 100  63 100  65 100  65 100

       *   *  32  91  64 100  65 100  66 100  66 100

       *   *  14  45  35 100  36 100  39 100  40 100

       *   *  21  90  48 100  49 100  51 100  52 100

       *   *  13  43  31 100  32 100  35 100  36 100

       *   *  10  44  24 100  25 100  28 100  29 100

       *   *  11  42  26 100  27 100  30 100  31 100

       *   *  16  74  36 100  37 100  40 100  41 100

We turned on the logging option to show dataplane under severe load and it is logging it many times an hour.

[bmorris1]

Hello,

There is a current known issue with the 3k series on 7.1.2. I would advise you downgrade and wait for 7.1.3 or raise a support case for this.

Ben

[Transporter]

Hi Ben,

Thank you for coming back to me. Do you know if this issue listed somewhere in Palo Alto documents? Checking 7.1.2 release notes and cannot find anything.

Cheers

[bmorris1]

I don't think there is any documents about this at the moment. Your best bet is a support case as they can give you more info about it.

Ben

[BPry]

Out of curiosity; is there any way to be notified about issues such as this as soon as it happens? Palo Alto doesn't seem to do a very good job of notifying owners about potential issues, as this is the first time I've seen any mention of major issues with 7.1.2 with the 3k series. 

[jlavender · Accepted solution]

---

This is tracked under bug 94790 and should be fixed in version 7.1.3 which is expected to be released on the first week of July.

---

[TranceforLife]

Hi BPry,

A do agree with you here. Especially when it is quite a big issue, that's affecting system/operation a lot. 

But no documents or notifications  I am afraid. 

Thanks

[nextgenhappines]

I have had some success to ask our sales engineer to check if there are any reported P1, P2 issues on a specific release.  If you have paid for TAM service,  TAM should be proactively updating their clients as well.   Also,  I will check to see if there are any hotfixes released after  2 to 4 weeks a new release available.  When you try to open a new case, on the software version drop down, you will see panos 7.0.1-H1, H2...  that tells me if a hotfix availble.   At that point, I will reach out to our sales engineer to ask for more information.  

[TranceforLife]

Hello,

Would be nice to have a hot fix. Please let me know if you get more info on this.

Cheers

[BPry]

I'll have to check and see if we have TAM; I know that we have support and have access to a SE but I wasn't involved in the actual purchasing of it or ever informed really of anything to do with support other than the contact information for the SE. 

[Brandon_Wertz]

@BPry you only have a TAM if you have preimum plus service support

[Brandon_Wertz]

IMO Palo hasn't done a very good job with the stability of the entire 7.X.X train of code.  7.0.0 was completely deferred and 7.1.0 -.2 have been riddled with certificate/SSL issues among other things.  They could do a lot better,

All that being said if you're in the position to have the responsibility to own/manage the firewall service in your organization then it's your responsibility to ensure the code is right for your production enviornment.  Look at the new features being offered by the new code release then weigh that business requirement against potential risk of service impact if the upgrade doesn't go well.

Are there a lot of bugs identified in the current 7.1.2 release, 11 pages of bug-IDs in-fact.  That in-it of itself stands out as a reason to not upgrade to it.

Sure we should hold Palo accountable for providing a code release that creates a service impact, but at the end of the day we're the ones accountable to our employers for the service outage if we deploy a prorduct with 11 pages of bugs.

[nextgenhappines]

Let's try to be fair.  I would be mainly concern about the P1 and P2 bugs.   It is impossible to test all the config permutation.     It will be great if PAN has an open bug databases to allow clients to search for reported identy bug and status update.

E   

[pulukas]

I second the desire for a searchable bug database.  This makes migrations and upgrades easier to select the right version of code for the particular configuration applied.

This can also let us find the right upgrade path to fix an obvious bug we hit on our own without needing to open a ticket and wait in the queues for support to search for the bug.

[Srivastava]

Is this fixed for  PA - 5050 as well ?  I see similar issues with PA - 5050 since it is upgraded to 7.1.1 in May 2016.