topic Re: PA2020 High CPU utilization "useridd" 100% management plane in General Topics

PA2020 High CPU utilization "useridd" 100% management plane

FabioGarcia — Mon, 04 Feb 2013 20:37:49 GMT

Dears,

My PA2020 has 2 agent working identifying my AD users... but the mgnt plane is running 100% all day long...

Any suggestion ?

pls find below the show resources output....

PA2020 running OS 5.0.2

top - 18:26:05 up 6 days, 1:33, 1 user, load average: 10.26, 11.02, 12.17 <<<<<<<<<<<<<<<< !!!!!

Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie

Cpu(s): 51.9%us, 46.0%sy, 2.1%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 995872k total, 901792k used, 94080k free, 5996k buffers

Swap: 2212876k total, 647316k used, 1565560k free, 179620k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2373 root 20 0 209m 72m 63m S 140 7.5 10861:51 useridd<<<<<<<<<<<<<<<<< 140% CPU !!!!

21021 nobody 20 0 429m 51m 4808 S 37 5.3 329:34.26 appweb3

2042 root 30 10 4468 964 792 R 4 0.1 0:00.12 top

2371 root 20 0 651m 210m 4076 S 4 21.6 118:50.34 mgmtsrvr

1720 admin 20 0 4532 1164 912 R 1 0.1 0:02.64 top

2405 root 20 0 355m 89m 2192 S 1 9.2 48:59.31 logrcvr

2142 root 15 -5 39636 2920 1240 S 1 0.3 106:28.41 sysd

2151 root 30 10 40568 3644 1692 S 0 0.4 21:50.38 python

2408 root 20 0 247m 2480 1628 S 0 0.2 5:39.85 varrcvr

2415 root 20 0 141m 2640 1760 S 0 0.3 1:17.82 routed

1 root 20 0 1836 560 536 S 0 0.1 0:02.30 init

Re: PA2020 High CPU utilization "useridd" 100% management plane

sdurga — Mon, 04 Feb 2013 22:29:55 GMT

Hi,

I would request you to open a case with support. This information looks buggy, and we need to investigate this further. Please open a case and providing the necessary files for further investigation.

Thanks,

Sandeep T

Re: PA2020 High CPU utilization "useridd" 100% management plane

mitch — Mon, 04 Feb 2013 23:32:18 GMT

Same issue here. A 2050. Upgraded to 5.0.2 get rid of some DHCP problems and now the management CPU is locked at 100%

Re: PA2020 High CPU utilization "useridd" 100% management plane

gswcowboy — Mon, 04 Feb 2013 23:39:30 GMT

Hello Mitch, please call your PAN Support or your ASC for a live debug of your high cpu issue. We'll need to gather traces/core files from your unit to assist with investigation.

Regards,

Renato

Re: PA2020 High CPU utilization "useridd" 100% management plane

DaveM — Tue, 05 Feb 2013 14:51:27 GMT

I have the same problem with 5.0.2 (I have a open support ticket)

I'm seeing 100% on my PA2020 and PA500 and about 50% on the PA200's

Regards

Dave

PID USER	PR NI VIRT RES SHR S %CPU %MEM	TIME+ COMMAND
2356	20 0 230m 100m 64m S 196 10.3 22515:04 useridd
26227	30 10 57904 13m 4236 R	5 1.4 0:00.38 pan_logdb_index
26249	20 0 4468 1032 800 R	5 0.1 0:00.06 top
2132	15 -5 54168 14m 1292 S	4 1.5 217:52.36 sysd
2134	0 -20 32228 5916 1188 S	4 0.6 38:32.82 masterd_manager
	1	20 0 1836 560 536 S	0 0.1 0:02.42 init
	2	20 0	0	0	0 S	0 0.0 0:00.01 kthreadd
	3	RT 0	0	0	0 S	0 0.0 0:02.09 migration/0
	4	20 0	0	0	0 S	0 0.0 0:00.15 ksoftirqd/0

Re: PA2020 High CPU utilization "useridd" 100% management plane

cmcguire-coactive — Tue, 05 Feb 2013 15:55:28 GMT

Obviously there is an issue with UserID consuming massive resources on 5.0.2 (if not 5.0 generally).

PAN - Can we expect a hotfix release anytime soon to address this?

Re: PA2020 High CPU utilization "useridd" 100% management plane

jochristian — Tue, 05 Feb 2013 16:26:34 GMT

Same problem here with useridd using 100% cpu. PA-2050

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2254 root 20 0 209m 74m 65m S 132 7.6 9812:57 useridd

2291 nobody 20 0 234m 34m 5796 S 28 3.5 17:55.53 appweb3

2251 root 20 0 539m 233m 5356 S 5 24.0 89:41.90 mgmtsrvr

30070 root 30 10 137m 99m 5628 R 2 10.3 0:09.48 pan_report_gen

2106 root 20 0 165m 2624 1952 S 2 0.3 4:11.80 sysdagent

30693 nobody 20 0 26984 1312 924 R 2 0.1 0:00.06 sdb

30665 root 20 0 12440 2212 1916 S 2 0.2 0:00.05 wmic

Seems to be an common issue for most users with 2000 series?

I will open an case with support.

Jo Christian

Re: PA2020 High CPU utilization "useridd" 100% management plane

jay.martin — Tue, 05 Feb 2013 17:06:52 GMT

Same here - 2020's and 2050's as well as 4060's - had to take 2000's back to 5.0.1 as commits and FQDN refreshes started failing. 4060's still running 5.0.2, but userid consuming LOTS of cpu constantly, but I suppose the beefier platform is handling it. Already reported to PAN but yet to hear back on how they intend to fix.

Re: PA2020 High CPU utilization "useridd" 100% management plane

mikand — Tue, 05 Feb 2013 17:20:12 GMT

Do you use a dedicated pan-agent server or do you use the new builtin pan-agent feature of panos 5.x ?

Re: PA2020 High CPU utilization "useridd" 100% management plane

DaveM — Tue, 05 Feb 2013 17:22:05 GMT

I've got two dedicated pan-agent servers - I've also upgraded these to the latest version - made no difference.

Re: PA2020 High CPU utilization "useridd" 100% management plane

jochristian — Tue, 05 Feb 2013 17:34:35 GMT

Using both agentless userid and agent server here.

Support suggested a few steps to try if using agentless.

First disable agentless configuration... Commit... debug software restart user-id... Wait 5-10 min... Enable agentless configuration and then commit.

Did seem to fix the problem for now on our PA-2050.

Jo Christian

Re: PA2020 High CPU utilization "useridd" 100% management plane

BobW — Wed, 06 Feb 2013 03:19:15 GMT

I have a case open on this as well. PA is aware of it and is actively coding a fix. The timeline is unknown at this point, "will definitely be in 5.0.3" but and could be "weeks away". According to the folks I spoke with it is a serious enough issue there may be some sort of hotfix issued, but he could not state that for sure.

It has been causing quite a bit of issues on or PA500: Captive portal timeouts, problems with UserID mappings etc.

Given the lack of a timeline I rolled back to 5.0.1 which fixed it immediately.

Hope that helps,

Bob

Re: PA2020 High CPU utilization "useridd" 100% management plane

jochristian — Wed, 06 Feb 2013 10:40:07 GMT

Hello,

We are also seeing the same issue on an PA-5050 installation using 5.0.2.

Seems like this is a 5.0.2 issue then.

Jo Christian

Re: PA2020 High CPU utilization "useridd" 100% management plane

pg_itdept — Wed, 06 Feb 2013 13:47:29 GMT

We are also seeing this on our PA-500 with 5.0.2. We have attempted to upgrade the agent although this has made no difference.

Re: PA2020 High CPU utilization "useridd" 100% management plane

Retired Member — Wed, 06 Feb 2013 15:29:11 GMT

Same here... since 5.0.2 with PA-2050

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2317 root 20 0 197m 73m 63m S 101 7.5 3047:40 useridd

22375 root 30 10 90500 49m 4148 S 13 5.1 0:00.41 pan_logdb_index

2106 root 15 -5 40572 3688 1116 S 7 0.4 85:24.78 sysd

8931 root 30 10 3996 1380 1112 S 6 0.1 1:55.05 genindex.sh

22376 root 30 10 26984 1316 924 R 3 0.1 0:00.09 sdb

167 root 20 0 0 0 0 S 1 0.0 13:09.53 kswapd0

Re: PA2020 High CPU utilization "useridd" 100% management plane

cmcguire-coactive — Wed, 06 Feb 2013 18:12:33 GMT

We've had a great number of problems (other than just Mgmt CPU) with 5.0.2 on our PA-500's. It seems to me that 5.0.2 has a number of issues, but that they manifest most dramatically (based on the other posts in this thread) on the PA-500 appliances (CPU and Memory capacity would be my guess).

We're rolling back to 4.1.10 tonight on the 500's and 5.0.1 on the VM-100 appliance.

Re: PA2020 High CPU utilization "useridd" 100% management plane

Sebbb — Wed, 06 Feb 2013 19:50:50 GMT

Same problem with 5.0.2 and 3020.

Case opened.

Re: PA2020 High CPU utilization "useridd" 100% management plane

mitch — Wed, 06 Feb 2013 23:15:23 GMT

Actually 5.0.0 seemed fine as far as the CPU issue goes. Unfortunately the DHCP server was borked. It would simply stop granting leases.

5.0.2 is perfect w.r.t. the DHCP ;_)

Re: PA2020 High CPU utilization "useridd" 100% management plane

smfwadmin — Fri, 08 Feb 2013 16:10:16 GMT

Same issue here PA2050 Running 5.0.2

PID USER	PR NI VIRT RES SHR S %CPU %MEM	TIME+ COMMAND
2331	20 0 244m 75m 64m S 124 7.8 7094:50 useridd
2329	20 0 624m 258m 5668 S 13 26.6 406:35.06 mgmtsrvr
3672	30 10 4468 992 792 R	8 0.1 0:00.18 top
3647	30 10 46936 1980 1672 R	7 0.2 0:00.18 pan_logdb_index
3686	20 0 4468 1064 800 R	7 0.1 0:00.10 top
3696	20 0 12440 1996 1720 S	7 0.2 0:00.05 wmic

Re: PA2020 High CPU utilization "useridd" 100% management plane

FabioGarcia — Fri, 08 Feb 2013 16:55:46 GMT

Dears,

Yesterday we have moved back to 5.0.1 and everything is OK now.... unless the real time QoS graphical which is not OK... but that is a known bug for this version....

PA confirmed PANOS 5.0.2 has a bug for useridd process and there is no estimated time of correction... so I did that .. and I also suggest everyone to do the same.

Thanks in advance!!