CPU issues on PA-3410

[M.Sharma415844]

Hi Team,

Need some help with respect to below query:

Issue - We are using PA-3410 in our environment. We are monitoring this device using Logic Monitor tool. Our monitoring team states that they monitors CPU and memory utilization for Palo Alto devices using the “show system resources” command using API key instead of using SNMP OID.

Our monthly report gives data that PA-3410 is 80% utilized consistently. Based on the screenshot from monitoring tool, there is a parameter which is 80% utilized consistently.

Parameter is - "% CPU time spent in user space"

Observations - 

I have already reviewed the tech support file thoroughly and did not find any issues with CPU.

I would like to understand if there is a way to find out from the API KEY what this parameters (% CPU time spent in user space) is all about. I have attached a screenshot from my monitoring tool.

[reaper]

For the PA-3410 'show system resources' includes the dataplane pan_task processes, which are pre-spun up to 100%

Only cores 0, 10 and 11 are used by the management plane

top - 00:26:52 up 101 days, 19:29,  1 user,  load average: 9.33, 9.45, 9.49
Tasks: 303 total,  10 running, 293 sleeping,   0 stopped,   0 zombie
%Cpu0  :  2.3 us,  1.3 sy,  0.0 ni, 95.6 id,  0.0 wa,  0.3 hi,  0.3 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
%Cpu7  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
%Cpu8  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
%Cpu10 :  2.0 us,  1.4 sy,  0.0 ni, 96.3 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
%Cpu11 :  2.7 us,  1.3 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.3 hi,  0.3 si,  0.0 st
MiB Mem :  31767.4 total,    295.5 free,  18247.1 used,  13224.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9652.4 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 8718 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146249:23 pan_task
 8720 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146354:24 pan_task
 8721 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146350:21 pan_task
 8723 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146340:35 pan_task
 8724 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146343:29 pan_task
 8725 root      20   0  137.3g   5.1g   5.1g R 100.0  16.4 146344:55 pan_task
 8726 root      20   0  137.4g   5.1g   5.1g R 100.0  16.5 146351:31 pan_task
 8719 root      20   0  137.3g   5.1g   5.1g R  99.7  16.4 146287:52 pan_task
 8722 root      20   0  137.3g   5.1g   5.1g R  99.7  16.4 146350:23 pan_task
 8655 root      20   0       0      0      0 S   3.7   0.0   5200:11 kni_single
 6945 root      20   0  868624 258896  86184 S   1.3   0.8   1007:44 useridd
 8717 root      20   0 1613868  72172   9736 S   1.3   0.2   1393:07 wifclient
 4814 root      10 -10  129.9g  60464   8600 S   1.0   0.2   1984:40 brdagent
 6941 root      20   0  604984 100920  33368 S   0.7   0.3 513:54.53 distributord
 7109 root      20   0  737420 125028  33792 S   0.7   0.4 532:15.13 dnsproxyd
13796 nobody    20   0  872932  53656  19272 S   0.7   0.2  16:28.00 httpd
 4520 root       0 -20 9644012   5.1g   5.1g S   0.3  16.5 477:45.76 md_apps

[M.Sharma415844]

Thank you for your response @reaper .

In our scenario, this parameter "us" is showing high (around 80%), my understanding says, this will be an expected behavior as there is only CPU in PA-3410. And overall CPU usage with respect to parameter "us" will be an expected behavior.

I tried to find some additional details as per the below:

> us (user) which means Percentage of CPU time running user-initiated programs, such as running queries and reports.
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClUfCAK
> Here reports mean custom reports and predefined reports. I checked and no custom reports are running on the firewall.
> All pre-defined reports are enabled on the firewall which would be one of the causes of the high "us".

> I also compared the details with other platform firewalls like PA-220, PA-3220, and PA-5220 and below are the details:
- All pre-defined reports are also enabled on these firewalls.
- Value of "us" parameter is high (around 80%) in PA-220 and for PA-3410 as well because these devices have only one CPU.
- Value of "us" parameter is very minimum (less than 1%) in PA-3220 and for PA-5220 because these devices have dedicated CPU for Management Plane and Data Plane.
> Below are the number of CPUs and Core details.
PA-3220 - 2 CPU ( 1 mp, 1 dp )
PA-3410 - 1 CPU ( 3 cores for mp, 9 cores for DP)
PA-5220 - 2 CPU ( 1 mp, 1 dp )

So can you confirm if I can understand this as an expected behavior?

[reaper]

that article is correct in regards to chassis that have a completely separate management plane and dataplane (hardware are different motherboards basically)

the 3400 has the cpu's shared on the hardware

[peter.castle]

I find this interesting as I see the same level of User Space CPU Usage (80% +) (still high % on pan_task per core) on both our active and even our standby PA-460, but only after the upgrade from 10.2.7 to 10.2.9. Before, on 10.2.7, this it was 15-20%.

It hasn't really been having an issue performance wise but I am concerned, as if this were a typical linux system I was monitoring, high user space utilisation would mean applications were using high CPU continuously.