topic Re: Palo Alto PA-3050 100 per cent CPU in General Topics

Palo Alto PA-3050 100 % CPU

Transporter — Sat, 18 Jun 2016 15:00:14 GMT

These were upgraded from 7.0.6 to 7.1.2 on 31 May. Since then we are suffering from the data plane very frequently using 100% CPU.

For example:

admin@PA-3050-5(active)> show running resource-monitor hour

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:

core 0 1 2 3 4 5

avg max avg max avg max avg max avg max avg max

* * 7 35 16 100 17 100 19 100 20 100

* * 5 45 10 73 11 74 13 75 13 75

* * 4 15 8 60 8 60 10 61 10 60

* * 4 6 7 12 8 13 10 15 10 16

* * 4 6 7 19 8 19 9 22 10 22

* * 4 6 7 15 8 15 9 19 10 19

* * 4 7 7 13 8 14 10 16 10 17

* * 4 7 8 14 8 15 10 17 11 18

* * 24 89 45 100 46 100 48 100 48 100

* * 42 81 81 100 81 100 82 100 82 100

* * 41 85 85 100 85 100 86 100 86 100

* * 27 60 69 100 70 100 71 100 72 100

* * 35 79 77 100 77 100 78 100 78 100

* * 24 77 51 100 52 100 54 100 55 100

* * 28 79 59 100 59 100 61 100 62 100

* * 30 83 65 100 66 100 68 100 68 100

* * 29 96 63 100 63 100 65 100 65 100

* * 32 91 64 100 65 100 66 100 66 100

* * 14 45 35 100 36 100 39 100 40 100

* * 21 90 48 100 49 100 51 100 52 100

* * 13 43 31 100 32 100 35 100 36 100

* * 10 44 24 100 25 100 28 100 29 100

* * 11 42 26 100 27 100 30 100 31 100

* * 16 74 36 100 37 100 40 100 41 100

We turned on the logging option to show dataplane under severe load and it is logging it many times an hour.

Re: Palo Alto PA-3050 100 per cent CPU

bmorris1 — Thu, 16 Jun 2016 13:06:47 GMT

Hello,

There is a current known issue with the 3k series on 7.1.2. I would advise you downgrade and wait for 7.1.3 or raise a support case for this.

Ben

Re: Palo Alto PA-3050 100 per cent CPU

Transporter — Thu, 16 Jun 2016 13:18:47 GMT

Hi Ben,

Thank you for coming back to me. Do you know if this issue listed somewhere in Palo Alto documents? Checking 7.1.2 release notes and cannot find anything.

Cheers

Re: Palo Alto PA-3050 100 per cent CPU

bmorris1 — Thu, 16 Jun 2016 13:34:07 GMT

I don't think there is any documents about this at the moment. Your best bet is a support case as they can give you more info about it.

Ben

Re: Palo Alto PA-3050 100 per cent CPU

BPry — Thu, 16 Jun 2016 14:56:29 GMT

Out of curiosity; is there any way to be notified about issues such as this as soon as it happens? Palo Alto doesn't seem to do a very good job of notifying owners about potential issues, as this is the first time I've seen any mention of major issues with 7.1.2 with the 3k series.

Re: Palo Alto PA-3050 100 per cent CPU

jlavender — Fri, 17 Jun 2016 11:42:22 GMT

This is tracked under bug 94790 and should be fixed in version 7.1.3 which is expected to be released on the first week of July.

Re: Palo Alto PA-3050 100 per cent CPU

TranceforLife — Sat, 18 Jun 2016 14:58:00 GMT

Hi BPry,

A do agree with you here. Especially when it is quite a big issue, that's affecting system/operation a lot.

But no documents or notifications I am afraid.

Thanks

Re: Palo Alto PA-3050 100 per cent CPU

nextgenhappines — Sun, 19 Jun 2016 13:44:22 GMT

I have had some success to ask our sales engineer to check if there are any reported P1, P2 issues on a specific release. If you have paid for TAM service, TAM should be proactively updating their clients as well. Also, I will check to see if there are any hotfixes released after 2 to 4 weeks a new release available. When you try to open a new case, on the software version drop down, you will see panos 7.0.1-H1, H2... that tells me if a hotfix availble. At that point, I will reach out to our sales engineer to ask for more information.

Re: Palo Alto PA-3050 100 per cent CPU

TranceforLife — Sun, 19 Jun 2016 16:20:52 GMT

Hello,

Would be nice to have a hot fix. Please let me know if you get more info on this.

Cheers

Re: Palo Alto PA-3050 100 per cent CPU

BPry — Fri, 24 Jun 2016 14:30:53 GMT

I'll have to check and see if we have TAM; I know that we have support and have access to a SE but I wasn't involved in the actual purchasing of it or ever informed really of anything to do with support other than the contact information for the SE.

Re: Palo Alto PA-3050 100 per cent CPU

Brandon_Wertz — Mon, 27 Jun 2016 12:20:54 GMT

@BPry you only have a TAM if you have preimum plus service support

Re: Palo Alto PA-3050 100 per cent CPU

Brandon_Wertz — Mon, 27 Jun 2016 12:36:07 GMT

IMO Palo hasn't done a very good job with the stability of the entire 7.X.X train of code. 7.0.0 was completely deferred and 7.1.0 -.2 have been riddled with certificate/SSL issues among other things. They could do a lot better,

All that being said if you're in the position to have the responsibility to own/manage the firewall service in your organization then it's your responsibility to ensure the code is right for your production enviornment. Look at the new features being offered by the new code release then weigh that business requirement against potential risk of service impact if the upgrade doesn't go well.

Are there a lot of bugs identified in the current 7.1.2 release, 11 pages of bug-IDs in-fact. That in-it of itself stands out as a reason to not upgrade to it.

Sure we should hold Palo accountable for providing a code release that creates a service impact, but at the end of the day we're the ones accountable to our employers for the service outage if we deploy a prorduct with 11 pages of bugs.

Re: Palo Alto PA-3050 100 per cent CPU

nextgenhappines — Mon, 27 Jun 2016 13:59:22 GMT

Let's try to be fair. I would be mainly concern about the P1 and P2 bugs. It is impossible to test all the config permutation. It will be great if PAN has an open bug databases to allow clients to search for reported identy bug and status update.

Re: Palo Alto PA-3050 100 per cent CPU

pulukas — Sun, 03 Jul 2016 11:04:23 GMT

I second the desire for a searchable bug database. This makes migrations and upgrades easier to select the right version of code for the particular configuration applied.

This can also let us find the right upgrade path to fix an obvious bug we hit on our own without needing to open a ticket and wait in the queues for support to search for the bug.

Re: Palo Alto PA-3050 100 per cent CPU

Srivastava — Tue, 05 Jul 2016 01:23:33 GMT

Is this fixed for PA - 5050 as well ? I see similar issues with PA - 5050 since it is upgraded to 7.1.1 in May 2016.

Re: Palo Alto PA-3050 100 per cent CPU

TranceforLife — Tue, 05 Jul 2016 07:06:10 GMT

Hi,

I don't know if this is affecting a PA-5k series. I didn't see anybody here reported but what l can confirm that even after installing from 7.1.1 > 7.1.3 on the PA-3050 we still experienced same symptoms of high data plane CPU spikes as on . In the end, TAC found out a lot of "unknown-UDP" sessions.

> show session all filter application unknown-udp

Eliminating these seems to fixed the issue.

Cheers