useridd process is consuming 100% CPU on the PA-5250

[dtran]

PAN-OS is 9.1.10 running on PA-5250.

The useridd process is consuming 100% CPU:

Tasks: 313 total, 1 running, 309 sleeping, 0 stopped, 3 zombie
%Cpu(s): 2.8 us, 1.5 sy, 0.0 ni, 95.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32640128 total, 197252 free, 5921556 used, 26521320 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 26240004 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6789 root 20 0 713520 261008 133124 S 100.7 0.8 1491:47 useridd
7153 root 20 0 4843656 3.071g 25000 S 2.7 9.9 5301:45 logrcvr

The PA-5250 is 100% idle on weekends.  Is this normal?

[BPry]

@dtran 

This wouldn't be expected. I would simply try running debug software restart process user-id and restarting the process to see if it crawls back up again or not. 

[dtran]

This happens on ALL four pairs of cluster firewalls. 

[BPry]

@dtran,

That's an important detail that was left out. Regardless if you have already recycled the user-id process and have seen this go back up to using 100% of its allocated resources that isn't what I would expect and I would open a TAC case so that they can look into the issue with you. It's possible something was misconfigured on the firewall or the user-id agent and the process is getting stuck trying to process some information.

If you wanted to look at the logs yourself in the meantime you could attempt to see if any relevant errors are recorded in the process log files via less mp-log useridd.log but be aware that this is a very active file and you will see Warning messages that are completely irrelevant. 

[dtran]

@BPry: 

1- I only see this issue on all PA-5250 but not on PA-850.  All PA-5250 and PA-850 are pointing to the same User Agent (UA) servers.

2- I am aware of the useridd.log

I open a TAC case with PAN and the TAC engineer is telling me the issue is with the configuration, until I told him that it has the same configuration as the PA-850.  They are looking into it.

[AlwinThomas]

Is this issue fixed for you ? We are also seeing the same issue with PA-5250

[dtran]

I gave up on PAN TAC support because the support is completely useless.  Lot of times, I feel like I am dealing with a five years old who has no clue what he/she is doing.  About 10% of the times, I get a good engineer who can understand the issue I am facing with; however, 90% of the times, they are just clueless.

Enough of my ranting, I ended up rebooting the device after the case got nowhere with TAC support.  I've also upgraded to 9.1.11hf3 and 9.1.12 so it might have resolved the issue.

YMMV.

[dtran]

Just checked one of the firewalls and the issue is still there:

top - 16:44:11 up 206 days, 1:51, 1 user, load average: 1.12, 1.13, 1.14
Tasks: 313 total, 1 running, 311 sleeping, 0 stopped, 1 zombie
%Cpu0 : 1.0 us, 0.3 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 59.8 us, 40.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.7 us, 0.3 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu16 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu17 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu18 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu19 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu20 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu21 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu22 : 0.0 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu23 : 0.7 us, 0.7 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32640128 total, 187960 free, 6024212 used, 26427956 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 26133916 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6473 root 20 0 725852 265368 132752 S 100.7 0.8 199354:31 useridd
6835 root 20 0 4826292 3.086g 23240 S 3.0 9.9 10235:59 logrcvr

[dtran]

I don't expect PAN to be perfect but PAN claimed that version 9.1.11 should fix this issue, but it does NOT.  We're running version 9.1.12.