7.1 Dataplane CPU Utilization

[jslabaugh1]

I'm curious how many people out there have had high dataplane CPU utilization ever since updating to 7.1?  We updated to 7.1 so that we could decrypt additional ciphers and ever since updating we've had abnormally high dataplane CPU utilization which does not make any sense to me as we are nowhere near the stated maximum specifications on our 5050s.  Our dataplane CPU is consistently between 70 and 100 percent.  Right this second we're at 61,998 sessions and 83% dataplane CPU utilization.  

[BPry]

I would consider contacting TAC on this as this appears to be a higher priority. Personally I haven't had any noticable dataplane CPU increases since updating everything to 7.1.*. Exactly what version of 7.1 are you running?

[jslabaugh1]

Yeah, I opened a TAC case and it went nowhere.  

Are you doing SSL decryption?

I've been on several different 7.1.x releases but we're currently on 7.1.8.

I've noticed that there is almost always a session showing bad key when running the 'show running resource-monitor ingress-backlogs' command which makes me think that my dataplane CPU issues are due to decryption.

> show running resource-monitor ingress-backlogs

-- SLOT: s1, DP: dp0 --

USAGE - ATOMIC: 1% TOTAL: 3%

-- SLOT: s1, DP: dp1 --

USAGE - ATOMIC: 33% TOTAL: 34%

TOP SESSIONS:

SESS-ID         PCT     GRP-ID  COUNT

34018913        3%      7       73

34359173        3%      1       2

                        3       57

                        7       3

37554439        3%      12      72

SESSION DETAILS

SESS-ID         PROTO   SZONE   SRC             SPORT   DST             DPORT

IGR-IF          EGR-IF         TYPE     APP

34018913        6       INET_Inside     x.x.x.52   26774   65.158.47.64

80      ethernet1/14    ethernet1/13    FLOW    ms-update

34359173        6       WF_Inside       x.x.x.61   58354   134.119.52.47

443     ae2     ae3     FLOW    web-browsing

> show session id 37554439

Session        37554439

            Bad Key: c2s: 'c2s'

            Bad Key: s2c: 's2c'

        index(local):                        : 4000007

[jdprovine]

I am curious when  you say you opened a tac case and it went no where

[BPry]

Since you are adding ciphers it isn't out of the ordinarry to see the dataplane CPU go up, but it sounds like something in your initial 7.1 update didn't go so well and is causing the issue. I saw a small increase with the update on devices where we decrypt the traffic, again though it wasn't that noticable and it was to be expected.

@jdprovine,

I've noticed this a lot lately. I'll open a ticket and then sit through all the level 1 normal random stuff, then they ask for some additional logs and when I can't get it to reproduce or can't actually troubleshoot the steps that they want to do the case essentially becomes stale. I've stayed away from TAC for a while and simply communicate with my SE since he generally just puts me in contact with the proper person.

[jdprovine]

I currently do no have an assigned SE and in the past they have pointed me to tac and nto given me much support. I have to admit my last two tickets with tac have been less than stellar, I usually end up figuring it out myself

[sagarmanandhar]

was the issue solved, if yes can you tell?

[kiwi]

Hi @sagarmanandhar ,

I'm not convinced a 7y old discussion might still be relevant on current PAN-OS and hardware.

Could you explain the issue you're have so me might give some pointers ?

You might also want to check out the following Support FAQ on high DP CPU load:

https://live.paloaltonetworks.com/t5/support-faq/support-faq-how-to-handle-high-data-plane-cpu-issue...

Kind regards,

-Kim.