topic Re: CPU issues on PA-3410 in Next-Generation Firewall Discussions

CPU issues on PA-3410

M.Sharma415844 — Fri, 07 Jun 2024 02:19:07 GMT

Hi Team,

Need some help with respect to below query:

Issue - We are using PA-3410 in our environment. We are monitoring this device using Logic Monitor tool. Our monitoring team states that they monitors CPU and memory utilization for Palo Alto devices using the “show system resources” command using API key instead of using SNMP OID.

Our monthly report gives data that PA-3410 is 80% utilized consistently. Based on the screenshot from monitoring tool, there is a parameter which is 80% utilized consistently.

Parameter is - "% CPU time spent in user space"

Observations -

I have already reviewed the tech support file thoroughly and did not find any issues with CPU.

I would like to understand if there is a way to find out from the API KEY what this parameters (% CPU time spent in user space) is all about. I have attached a screenshot from my monitoring tool.

Re: CPU issues on PA-3410

reaper — Fri, 07 Jun 2024 07:28:09 GMT

For the PA-3410 'show system resources' includes the dataplane pan_task processes, which are pre-spun up to 100%

Only cores 0, 10 and 11 are used by the management plane

top - 00:26:52 up 101 days, 19:29, 1 user, load average: 9.33, 9.45, 9.49 Tasks: 303 total, 10 running, 293 sleeping, 0 stopped, 0 zombie %Cpu0 : 2.3 us, 1.3 sy, 0.0 ni, 95.6 id, 0.0 wa, 0.3 hi, 0.3 si, 0.0 st %Cpu1 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 99.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st %Cpu7 : 99.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st %Cpu8 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu9 : 99.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st %Cpu10 : 2.0 us, 1.4 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st %Cpu11 : 2.7 us, 1.3 sy, 0.0 ni, 95.3 id, 0.0 wa, 0.3 hi, 0.3 si, 0.0 st MiB Mem : 31767.4 total, 295.5 free, 18247.1 used, 13224.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 9652.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8718 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146249:23 pan_task 8720 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146354:24 pan_task 8721 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146350:21 pan_task 8723 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146340:35 pan_task 8724 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146343:29 pan_task 8725 root 20 0 137.3g 5.1g 5.1g R 100.0 16.4 146344:55 pan_task 8726 root 20 0 137.4g 5.1g 5.1g R 100.0 16.5 146351:31 pan_task 8719 root 20 0 137.3g 5.1g 5.1g R 99.7 16.4 146287:52 pan_task 8722 root 20 0 137.3g 5.1g 5.1g R 99.7 16.4 146350:23 pan_task 8655 root 20 0 0 0 0 S 3.7 0.0 5200:11 kni_single 6945 root 20 0 868624 258896 86184 S 1.3 0.8 1007:44 useridd 8717 root 20 0 1613868 72172 9736 S 1.3 0.2 1393:07 wifclient 4814 root 10 -10 129.9g 60464 8600 S 1.0 0.2 1984:40 brdagent 6941 root 20 0 604984 100920 33368 S 0.7 0.3 513:54.53 distributord 7109 root 20 0 737420 125028 33792 S 0.7 0.4 532:15.13 dnsproxyd 13796 nobody 20 0 872932 53656 19272 S 0.7 0.2 16:28.00 httpd 4520 root 0 -20 9644012 5.1g 5.1g S 0.3 16.5 477:45.76 md_apps

Re: CPU issues on PA-3410

M.Sharma415844 — Tue, 18 Jun 2024 03:52:07 GMT

Thank you for your response @reaper .

In our scenario, this parameter "us" is showing high (around 80%), my understanding says, this will be an expected behavior as there is only CPU in PA-3410. And overall CPU usage with respect to parameter "us" will be an expected behavior.

I tried to find some additional details as per the below:

> us (user) which means Percentage of CPU time running user-initiated programs, such as running queries and reports.
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClUfCAK
> Here reports mean custom reports and predefined reports. I checked and no custom reports are running on the firewall.
> All pre-defined reports are enabled on the firewall which would be one of the causes of the high "us".

> I also compared the details with other platform firewalls like PA-220, PA-3220, and PA-5220 and below are the details:
- All pre-defined reports are also enabled on these firewalls.
- Value of "us" parameter is high (around 80%) in PA-220 and for PA-3410 as well because these devices have only one CPU.
- Value of "us" parameter is very minimum (less than 1%) in PA-3220 and for PA-5220 because these devices have dedicated CPU for Management Plane and Data Plane.
> Below are the number of CPUs and Core details.
PA-3220 - 2 CPU ( 1 mp, 1 dp )
PA-3410 - 1 CPU ( 3 cores for mp, 9 cores for DP)
PA-5220 - 2 CPU ( 1 mp, 1 dp )

So can you confirm if I can understand this as an expected behavior?

Re: CPU issues on PA-3410

reaper — Tue, 18 Jun 2024 14:14:37 GMT

that article is correct in regards to chassis that have a completely separate management plane and dataplane (hardware are different motherboards basically)

the 3400 has the cpu's shared on the hardware

Re: CPU issues on PA-3410

peter.castle — Tue, 22 Oct 2024 02:42:35 GMT

I find this interesting as I see the same level of User Space CPU Usage (80% +) (still high % on pan_task per core) on both our active and even our standby PA-460, but only after the upgrade from 10.2.7 to 10.2.9. Before, on 10.2.7, this it was 15-20%.

It hasn't really been having an issue performance wise but I am concerned, as if this were a typical linux system I was monitoring, high user space utilisation would mean applications were using high CPU continuously.