PA2020 High CPU utilization "useridd" 100% management plane

[FabioGarcia]

Dears,

My PA2020 has 2 agent working identifying my AD users... but the mgnt plane is running 100% all day long...

Any suggestion ?

pls find below the show resources output....

PA2020 running OS 5.0.2

top - 18:26:05 up 6 days,  1:33,  1 user, load average: 10.26, 11.02, 12.17  <<<<<<<<<<<<<<<< !!!!!

Tasks: 100 total,   2 running,  98 sleeping,   0 stopped,   0 zombie

Cpu(s): 51.9%us, 46.0%sy,  2.1%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:    995872k total,   901792k used,    94080k free,     5996k buffers

Swap:  2212876k total,   647316k used,  1565560k free,   179620k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

2373 root      20   0  209m  72m  63m S  140  7.5  10861:51 useridd<<<<<<<<<<<<<<<<< 140% CPU !!!!

21021 nobody    20   0  429m  51m 4808 S   37  5.3 329:34.26 appweb3

2042 root      30  10  4468  964  792 R    4  0.1   0:00.12 top

2371 root      20   0  651m 210m 4076 S    4 21.6 118:50.34 mgmtsrvr

1720 admin     20   0  4532 1164  912 R    1  0.1   0:02.64 top

2405 root      20   0  355m  89m 2192 S    1  9.2  48:59.31 logrcvr

2142 root      15  -5 39636 2920 1240 S    1  0.3 106:28.41 sysd

2151 root      30  10 40568 3644 1692 S    0  0.4  21:50.38 python

2408 root      20   0  247m 2480 1628 S    0  0.2   5:39.85 varrcvr

2415 root      20   0  141m 2640 1760 S    0  0.3   1:17.82 routed

    1 root      20   0  1836  560  536 S    0  0.1   0:02.30 init

[sdurga]

Hi,

I would request you to open a case with support. This information looks buggy, and we need to investigate this further. Please open a case and providing the necessary files for further investigation.

Thanks,

Sandeep T

[mitch]

Same issue here. A 2050. Upgraded to 5.0.2 get rid of some DHCP problems and now the management CPU is locked at 100%

[gswcowboy]

Hello Mitch, please call your PAN Support or your ASC for a live debug of your high cpu issue. We'll need to gather traces/core files from your unit to assist with investigation.

Regards,

Renato   

[DaveM]

I have the same problem with 5.0.2 (I have a open support ticket)

I'm seeing 100% on my PA2020 and PA500 and about 50% on the PA200's

Regards

Dave

PID USER	PR  NI  VIRT  RES  SHR S %CPU %MEM	TIME+  COMMAND
2356	20   0  230m 100m  64m S  196 10.3  22515:04 useridd
26227	30  10 57904  13m 4236 R	5  1.4   0:00.38 pan_logdb_index
26249	20   0  4468 1032  800 R	5  0.1   0:00.06 top
2132	15  -5 54168  14m 1292 S	4  1.5 217:52.36 sysd
2134	0 -20 32228 5916 1188 S	4  0.6  38:32.82 masterd_manager
	1	20   0  1836  560  536 S	0  0.1   0:02.42 init
	2	20   0	0	0	0 S	0  0.0   0:00.01 kthreadd
	3	RT   0	0	0	0 S	0  0.0   0:02.09 migration/0
	4	20   0	0	0	0 S	0  0.0   0:00.15 ksoftirqd/0

[coactive]

Obviously there is an issue with UserID consuming massive resources on 5.0.2 (if not 5.0 generally).

PAN - Can we expect a hotfix release anytime soon to address this?

[jochristian]

Same problem here with useridd using 100% cpu. PA-2050

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

2254 root      20   0  209m  74m  65m S  132  7.6   9812:57 useridd

2291 nobody    20   0  234m  34m 5796 S   28  3.5  17:55.53 appweb3

2251 root      20   0  539m 233m 5356 S    5 24.0  89:41.90 mgmtsrvr

30070 root      30  10  137m  99m 5628 R    2 10.3   0:09.48 pan_report_gen

2106 root      20   0  165m 2624 1952 S    2  0.3   4:11.80 sysdagent

30693 nobody    20   0 26984 1312  924 R    2  0.1   0:00.06 sdb

30665 root      20   0 12440 2212 1916 S    2  0.2   0:00.05 wmic

Seems to be an common issue for most users with 2000 series?

I will open an case with support.

Jo Christian

[jay.martin]

Same here - 2020's and 2050's as well as 4060's - had to take 2000's back to 5.0.1 as commits and FQDN refreshes started failing.  4060's still running 5.0.2, but userid consuming LOTS of cpu constantly, but I suppose the beefier platform is handling it.  Already reported to PAN but yet to hear back on how they intend to fix.

[mikand]

Do you use a dedicated pan-agent server or do you use the new builtin pan-agent feature of panos 5.x ?

[DaveM]

I've got two dedicated pan-agent servers - I've also upgraded these to the latest version - made no difference.

[jochristian]

Using both agentless userid and agent server here.

Support suggested a few steps to try if using agentless.

First disable agentless configuration... Commit... debug software restart user-id... Wait 5-10 min... Enable agentless configuration and then commit.

Did seem to fix the problem for now on our PA-2050.

Jo Christian

[BobW]

I have a case open on this as well.  PA is aware of it and is actively coding a fix.  The timeline is unknown at this point, "will definitely be in 5.0.3" but and could be "weeks away".  According to the folks I spoke with it is a serious enough issue there may be some sort of hotfix issued, but he could not state that for sure.

It has been causing quite a bit of issues on or PA500:  Captive portal timeouts, problems with UserID mappings etc.

Given the lack of a timeline I rolled back to 5.0.1 which fixed it immediately.

Hope that helps,

Bob

[jochristian]

Hello,

We are also seeing the same issue on an PA-5050 installation using 5.0.2.

Seems like this is a 5.0.2 issue then.

Jo Christian

[pg_itdept]

We are also seeing this on our PA-500 with 5.0.2. We have attempted to upgrade the agent although this has made no difference.

[pboegli]

Same here... since 5.0.2 with PA-2050

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

2317 root      20   0  197m  73m  63m S  101  7.5   3047:40 useridd

22375 root      30  10 90500  49m 4148 S   13  5.1   0:00.41 pan_logdb_index

2106 root      15  -5 40572 3688 1116 S    7  0.4  85:24.78 sysd

8931 root      30  10  3996 1380 1112 S    6  0.1   1:55.05 genindex.sh

22376 root      30  10 26984 1316  924 R    3  0.1   0:00.09 sdb

  167 root      20   0     0    0    0 S    1  0.0  13:09.53 kswapd0