topic High CPU on the management plane in General Topics

High CPU on the management plane

dtran — Sat, 11 Sep 2021 19:32:41 GMT

I am running 9.1.10 on the PA-850. Last night, I replaced the User-Agent Server and pointed the PA-850 to the new UA server. After that, the CPU on the management goes up to 100% and stayed there until I had to reboot the PA-850. I attempted to restart the management server process but that didn't fix it either. After reboot, the CPU on the management plane goes down to about 20%.

I also pointed three different PAN firewalls (PA-5250) to the new UA server last night and didn't experience any high CPU issue.

Is that normal? I opened a TAC case with PAN support but I am not expecting much from them.

Even after the reboot, I am seeing pan_task taking up 100% CPU:

Tasks: 163 total, 6 running, 157 sleeping, 0 stopped, 0 zombie
%Cpu(s): 66.1 us, 1.7 sy, 0.2 ni, 31.8 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 4119484 total, 647268 free, 1693244 used, 1778972 buff/cache
KiB Swap: 6104084 total, 6098908 free, 5176 used. 1310312 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3436 20 0 76376 36876 9128 R 100.0 0.9 87:06.78 pan_task
3438 20 0 51368 11700 9048 R 100.0 0.3 87:06.87 pan_task
3440 20 0 51368 11728 9068 R 100.0 0.3 87:06.58 pan_task
3437 20 0 51368 11628 8980 R 94.4 0.3 87:06.87 pan_task
3439 20 0 51428 11776 9080 R 94.4 0.3 87:06.64 pan_task
4468 20 0 1487740 31776 17672 S 11.1 0.8 0:06.05 varrcvr

Has anyone run into this issue before?

Re: High CPU on the management plane

BPry — Sun, 12 Sep 2021 03:16:37 GMT

@dtran,

I wouldn't worry about pan_task displaying a higher CPU percentage. It's only allocated a set amount of resources, and of the allocated resources pan_task processes will usually be anywhere in that 80%-100% of allocated resources available to the process.

Re: High CPU on the management plane

aleksandar.astardzhiev — Mon, 13 Sep 2021 08:29:04 GMT

Hi @dtran ,

To add more details I will quote explanation from TAC engineer, that received the first time I was dealing with PA-8xx firewall.

The PA-820 has 8 cores, three dedicated for MP usage (0,6,7) and four for DP usage (1,2,3,4). Core 5 is unused for PA-820, as it is reserved for DP usage for PA-850 platforms. The pan_task process is responsible for packet processing on the dataplanes and is always running and consuming all available CPU from the dedicated DP cores, so this will not take away any from CPU available for management usage (such as commits). As we can see that cores 6 and 7 are currently at 0% and core 0 is at 4.2% usage I would consider it safe to say that there is no MP CPU problem on this firewall. The difference between those platforms [PA-3xxx and PA-5xxx ] and PA-8XX is that they have a dedicated dataplane. For platforms like PA-8XX and PA-2XX with one CPU there are always some cores dedicated to MP and others for DP, and the DP cores of that single CPU are generally maxed out.

Re: High CPU on the management plane

dtran — Mon, 13 Sep 2021 10:58:37 GMT

Well, in my situation, CPUs 1,2,3,4,5 are 0% idle. The firewall has almost no traffics passing through. Is that expected behavior?

top - 10:54:47 up 1 day, 16:56, 1 user, load average: 5.10, 5.18, 5.22
Tasks: 160 total, 6 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 99.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu6 : 0.7 us, 0.7 sy, 0.3 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.7 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 4119484 total, 525540 free, 1829344 used, 1764600 buff/cache
KiB Swap: 6104084 total, 6071988 free, 32096 used. 1340340 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3437 root 20 0 51368 11972 9232 R 100.0 0.3 2454:03 pan_task
3436 root 20 0 76548 36940 8960 R 99.7 0.9 2454:05 pan_task
3438 root 20 0 51712 12092 9024 R 99.7 0.3 2454:05 pan_task
3439 root 20 0 51600 12264 9384 R 99.7 0.3 2454:02 pan_task
3440 root 20 0 51540 11956 9108 R 99.7 0.3 2454:03 pan_task
4309 root 20 0 545608 229600 131068 S 0.7 5.6 26:08.35 useridd