02-14-2013 07:30 PM
I have a PAN fw at a client site that always has the mgmt plane cpu at 100%. The data plane CPU barely ever goes above 10%. Is there an easy way to troubleshoot the cause of this or what is taking up so much CPU usage?
02-15-2013 05:47 AM
02-14-2013 09:30 PM
Which software version are your running on the firewall ? also can you please give me the output of the command "show system resources"
Thanks,
Sandeep T
02-15-2013 05:39 AM
Thanks for offering to help! This is a PAN 500 running 5.0.2. I disabled most of my application identification and ssl decryption to see if it would help yesterday but it did not. Here is the output:
top - 07:38:52 up 24 days, 13:19, 1 user, load average: 9.40, 8.21, 8.05
Tasks: 105 total, 2 running, 101 sleeping, 1 stopped, 1 zombie
Cpu(s): 14.0%us, 23.5%sy, 8.9%ni, 53.2%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 995872k total, 917440k used, 78432k free, 19900k buffers
Swap: 2008084k total, 327700k used, 1680384k free, 161728k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2473 20 0 190m 65m 63m S 175 6.7 22999:50 useridd
1701 30 10 46936 5384 2872 R 9 0.5 0:00.28 pan_logdb_index
1705 20 0 4468 1032 800 R 3 0.1 0:00.04 top
2508 20 0 97656 1520 1160 S 2 0.2 18:44.14 keymgr
1 20 0 1836 564 536 S 0 0.1 0:00.97 init
2 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 RT 0 0 0 0 S 0 0.0 0:12.55 migration/0
4 20 0 0 0 0 S 0 0.0 0:00.62 ksoftirqd/0
5 RT 0 0 0 0 S 0 0.0 0:15.73 migration/1
6 20 0 0 0 0 S 0 0.0 0:00.68 ksoftirqd/1
7 20 0 0 0 0 S 0 0.0 2:00.95 events/0
8 20 0 0 0 0 S 0 0.0 1:27.66 events/1
9 20 0 0 0 0 S 0 0.0 0:00.04 khelper
12 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr
112 20 0 0 0 0 S 0 0.0 0:00.01 sync_supers
114 20 0 0 0 0 S 0 0.0 0:00.01 bdi-default
115 20 0 0 0 0 S 0 0.0 0:05.13 kblockd/0
116 20 0 0 0 0 S 0 0.0 0:01.88 kblockd/1
125 20 0 0 0 0 S 0 0.0 0:00.00 ata/0
126 20 0 0 0 0 S 0 0.0 0:00.00 ata/1
127 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux
132 20 0 0 0 0 S 0 0.0 0:00.00 khubd
135 20 0 0 0 0 S 0 0.0 0:00.00 kseriod
156 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/0
157 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/1
172 20 0 0 0 0 S 0 0.0 8:53.86 kswapd0
173 20 0 0 0 0 S 0 0.0 0:00.00 aio/0
174 20 0 0 0 0 S 0 0.0 0:00.00 aio/1
175 20 0 0 0 0 S 0 0.0 0:00.00 nfsiod
732 20 0 0 0 0 S 0 0.0 0:00.39 octeon-ethernet
760 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
765 20 0 0 0 0 S 0 0.0 0:00.96 mtdblockd
793 20 0 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
833 20 0 0 0 0 S 0 0.0 0:39.21 kjournald
886 16 -4 1996 404 400 S 0 0.0 0:01.19 udevd
1595 20 0 21340 2472 2040 S 0 0.2 0:00.13 sshd
1628 ksaadmin 20 0 21340 1480 1036 S 0 0.1 0:00.00 sshd
1629 ksaadmin 20 0 97744 22m 11m S 0 2.3 0:03.20 cli
1702 ksaadmin 20 0 2976 668 564 S 0 0.1 0:00.05 less
1704 20 0 3832 1192 1056 S 0 0.1 0:00.06 sh
1706 20 0 1940 540 464 S 0 0.1 0:00.01 sed
1843 20 0 0 0 0 S 0 0.0 0:01.09 kjournald
1844 20 0 0 0 0 S 0 0.0 0:00.00 kjournald
02-15-2013 05:44 AM
top - 07:43:30 up 24 days, 13:24, 1 user, load average: 8.62, 8.59, 8.26
Tasks: 104 total, 1 running, 101 sleeping, 1 stopped, 1 zombie
Cpu(s): 14.0%us, 23.5%sy, 8.9%ni, 53.2%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 995872k total, 928432k used, 67440k free, 23456k buffers
Swap: 2008084k total, 326932k used, 1681152k free, 166964k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2473 20 0 190m 65m 63m S 179 6.7 23008:07 useridd
3284 20 0 4468 1032 800 R 3 0.1 0:00.05 top
19080 30 10 3896 1344 1104 S 3 0.1 1:03.09 genindex.sh
2288 0 -20 65244 4796 2048 S 2 0.5 47:29.27 masterd_core
2305 15 -5 36820 2192 1256 S 2 0.2 331:09.30 sysd
2506 20 0 308m 53m 1992 S 2 5.5 6:34.11 logrcvr
1 20 0 1836 564 536 S 0 0.1 0:00.97 init
2 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 RT 0 0 0 0 S 0 0.0 0:12.55 migration/0
4 20 0 0 0 0 S 0 0.0 0:00.62 ksoftirqd/0
5 RT 0 0 0 0 S 0 0.0 0:15.73 migration/1
6 20 0 0 0 0 S 0 0.0 0:00.68 ksoftirqd/1
7 20 0 0 0 0 S 0 0.0 2:00.96 events/0
8 20 0 0 0 0 S 0 0.0 1:27.67 events/1
9 20 0 0 0 0 S 0 0.0 0:00.04 khelper
12 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr
112 20 0 0 0 0 S 0 0.0 0:00.01 sync_supers
114 20 0 0 0 0 S 0 0.0 0:00.01 bdi-default
115 20 0 0 0 0 S 0 0.0 0:05.13 kblockd/0
116 20 0 0 0 0 S 0 0.0 0:01.88 kblockd/1
125 20 0 0 0 0 S 0 0.0 0:00.00 ata/0
126 20 0 0 0 0 S 0 0.0 0:00.00 ata/1
127 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux
132 20 0 0 0 0 S 0 0.0 0:00.00 khubd
135 20 0 0 0 0 S 0 0.0 0:00.00 kseriod
156 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/0
157 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/1
172 20 0 0 0 0 S 0 0.0 8:53.86 kswapd0
173 20 0 0 0 0 S 0 0.0 0:00.00 aio/0
174 20 0 0 0 0 S 0 0.0 0:00.00 aio/1
175 20 0 0 0 0 S 0 0.0 0:00.00 nfsiod
732 20 0 0 0 0 S 0 0.0 0:00.39 octeon-ethernet
760 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
765 20 0 0 0 0 S 0 0.0 0:00.96 mtdblockd
793 20 0 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
833 20 0 0 0 0 S 0 0.0 0:39.23 kjournald
886 16 -4 1996 404 400 S 0 0.0 0:01.19 udevd
1843 20 0 0 0 0 S 0 0.0 0:01.09 kjournald
1844 20 0 0 0 0 S 0 0.0 0:00.00 kjournald
1973 20 0 0 0 0 S 0 0.0 1:11.37 flush-8:0
2036 20 0 2008 628 572 S 0 0.1 0:00.99 syslogd
2039 20 0 1892 332 328 S 0 0.0 0:00.01 klogd
2048 20 0 1872 328 236 S 0 0.0 0:00.84 irqbalance
2056 rpc 20 0 2084 492 488 S 0 0.0 0:00.01 portmap
2074 20 0 2116 652 648 S 0 0.1 0:00.06 rpc.statd
2143 20 0 6868 744 640 S 0 0.1 0:00.01 sshd
2191 20 0 6804 432 428 S 0 0.0 0:00.01 sshd
2200 20 0 3280 596 592 S 0 0.1 0:00.01 xinetd
2219 20 0 0 0 0 S 0 0.0 0:00.00 lockd
2220 20 0 0 0 0 S 0 0.0 2:13.63 nfsd
2221 20 0 0 0 0 S 0 0.0 2:09.43 nfsd
2222 20 0 0 0 0 S 0 0.0 2:05.24 nfsd
2223 20 0 0 0 0 S 0 0.0 2:05.25 nfsd
2224 20 0 0 0 0 S 0 0.0 2:08.65 nfsd
2225 20 0 0 0 0 S 0 0.0 2:08.85 nfsd
2226 20 0 0 0 0 S 0 0.0 2:11.04 nfsd
2227 20 0 0 0 0 S 0 0.0 2:13.55 nfsd
2230 20 0 2488 688 580 S 0 0.1 0:03.51 rpc.mountd
2291 20 0 1888 456 452 S 0 0.0 0:00.00 agetty
2298 0 -20 27864 1508 1128 S 0 0.2 5:34.42 masterd_manager
2307 0 -20 32356 5180 1148 S 0 0.5 79:07.52 masterd_manager
2313 20 0 91736 3328 2064 S 0 0.3 0:03.03 dagger
2314 30 10 40568 3784 1808 S 0 0.4 62:58.04 python
2315 20 0 77772 2912 1644 S 0 0.3 0:07.70 cryptod
2316 20 0 166m 1868 1260 S 0 0.2 6:58.26 sysdagent
2321 20 0 7212 1100 1096 S 0 0.1 0:00.09 tscat
2333 20 0 71580 1168 1020 S 0 0.1 0:20.37 brdagent
2334 20 0 31912 1208 1040 S 0 0.1 1:07.53 ehmon
2335 20 0 47496 1180 1044 S 0 0.1 0:01.43 chasd
2422 20 0 0 0 0 S 0 0.0 0:02.82 kjournald
2463 20 0 2896 624 572 S 0 0.1 0:00.62 crond
2471 20 0 579m 304m 4256 S 0 31.3 71:53.12 mgmtsrvr
2496 20 0 214m 56m 8280 S 0 5.8 28:22.07 devsrvr
2505 20 0 91692 2916 2064 S 0 0.3 1:16.70 ikemgr
2507 20 0 99776 2540 1880 S 0 0.3 0:03.63 rasmgr
2508 20 0 97656 1520 1160 S 0 0.2 18:46.81 keymgr
2509 20 0 247m 2484 1652 S 0 0.2 3:30.72 varrcvr
2510 17 -3 56644 1848 1324 S 0 0.2 4:56.59 ha_agent
2511 20 0 111m 7856 1664 S 0 0.8 0:31.33 satd
2512 20 0 101m 2152 1428 S 0 0.2 0:07.89 sslmgr
2513 20 0 57132 1868 1464 S 0 0.2 0:02.34 dhcpd
2514 20 0 74608 2472 1528 S 0 0.2 0:03.04 dnsproxyd
2515 20 0 74388 1836 1460 S 0 0.2 0:03.78 pppoed
2517 20 0 142m 2604 1500 S 0 0.3 6:37.54 routed
2518 20 0 129m 4632 3640 S 0 0.5 0:14.07 authd
2973 20 0 21340 2476 2044 S 0 0.2 0:00.13 sshd
3028 20 0 27260 1368 1184 S 0 0.1 0:02.31 snmpd
3078 nobody 20 0 254m 59m 4120 S 0 6.1 69:26.59 appweb3
3147 ksaadmin 20 0 21340 1480 1036 S 0 0.1 0:00.00 sshd
3148 ksaadmin 20 0 97744 22m 11m S 0 2.3 0:03.12 cli
3280 ksaadmin 20 0 2976 668 564 S 0 0.1 0:00.03 less
3282 20 0 3832 1192 1056 S 0 0.1 0:00.07 sh
3285 20 0 1940 536 464 S 0 0.1 0:00.01 sed
4339 nobody 20 0 154m 10m 3304 S 0 1.0 2:31.39 appweb3
4357 nobody 20 0 180m 6516 2420 S 0 0.7 1:42.07 appweb3
18016 20 0 3980 3848 2956 S 0 0.4 0:00.03 ntpd
18868 30 10 0 0 0 Z 0 0.0 0:00.02 sh <defunct>
19031 20 0 2960 492 412 S 0 0.0 0:00.00 crond
19032 20 0 3720 1092 976 S 0 0.1 0:00.02 genindex_batch.
19039 20 0 33704 5040 2940 S 0 0.5 0:00.34 masterd_batch
26949 30 10 62864 26m 1884 T 0 2.7 0:27.14 bot_monitor.py
28754 20 0 2960 440 400 S 0 0.0 0:00.00 crond
28755 20 0 3720 852 772 S 0 0.1 0:00.00 bash
28756 20 0 33704 3176 1956 S 0 0.3 0:00.59 masterd_batch
02-15-2013 06:28 AM
Thanks for the information! I can't beat up PAN because I love their devices and resell them like crazy but it seems like every other software version has a quirk. 
02-15-2013 07:24 AM
Do you know if this will be address in 5.0.3?
My client uses QOS and the statistics graphing didn't work correctly in 5.0.1 so I upgraded them to 5.0.2. I would prefer not to downgrade back to 4.x.x as others have done.
02-15-2013 01:25 PM
A fix is currently being worked on and is being "targeted" for PAN OS 5.0.3.
02-19-2013 08:57 PM
This issue seems to be present in 4.1.11 as well, which is disappointing, as we rolled back from 5.0.2 to 4.1.10 to get away from this.
02-20-2013 06:51 AM
Thank you everyone for sharing this information.
I have recently downloaded 4.1.11. Before applying it, I browsed this forum to see if there were any known issues. From what I am reading, I believe I will stay on 4.1.10 and wait for 4.1.12.
I have an issue where committing a policy will generate errors and fail about 50% of the time. Subsequent commits with no changes eventually work. I am ever hopeful that this issue will be addressed in a new release.
02-25-2013 09:01 AM
I have the same problem:
PAN OS release 5.0.2
top - 17:59:34 up 3 days, 9:19, 1 user, load average: 7.70, 7.47, 7.32
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 27.1%us, 24.2%sy, 18.8%ni, 27.9%id, 1.8%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 995872k total, 934576k used, 61296k free, 14248k buffers
Swap: 2008084k total, 169024k used, 1839060k free, 246348k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2476 20 0 183m 72m 64m S 183 7.4 4347:37 useridd
27445 30 10 62036 21m 4272 R 15 2.3 0:02.21 pan_logdb_index
27453 20 0 4468 1028 800 R 5 0.1 0:00.06 top
1 20 0 1836 556 528 S 0 0.1 0:00.96 init
2 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 RT 0 0 0 0 S 0 0.0 0:01.53 migration/0
4 20 0 0 0 0 S 0 0.0 0:00.07 ksoftirqd/0
5 RT 0 0 0 0 S 0 0.0 0:01.78 migration/1
6 20 0 0 0 0 S 0 0.0 0:00.03 ksoftirqd/1
7 20 0 0 0 0 S 0 0.0 0:13.57 events/0
8 20 0 0 0 0 S 0 0.0 0:13.38 events/1
9 20 0 0 0 0 S 0 0.0 0:00.05 khelper
12 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr
112 20 0 0 0 0 S 0 0.0 0:00.00 sync_supers
114 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default
115 20 0 0 0 0 S 0 0.0 0:02.45 kblockd/0
116 20 0 0 0 0 S 0 0.0 0:01.62 kblockd/1
125 20 0 0 0 0 S 0 0.0 0:00.00 ata/0
126 20 0 0 0 0 S 0 0.0 0:00.00 ata/1
127 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux
132 20 0 0 0 0 S 0 0.0 0:00.00 khubd
135 20 0 0 0 0 S 0 0.0 0:00.00 kseriod
156 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/0
157 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/1
172 20 0 0 0 0 S 0 0.0 11:01.97 kswapd0
173 20 0 0 0 0 S 0 0.0 0:00.00 aio/0
174 20 0 0 0 0 S 0 0.0 0:00.00 aio/1
175 20 0 0 0 0 S 0 0.0 0:00.00 nfsiod
732 20 0 0 0 0 S 0 0.0 0:00.10 octeon-ethernet
760 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
765 20 0 0 0 0 S 0 0.0 0:00.97 mtdblockd
793 20 0 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
833 20 0 0 0 0 S 0 0.0 0:06.93 kjournald
886 16 -4 1996 440 400 S 0 0.0 0:01.15 udevd
1850 20 0 0 0 0 S 0 0.0 0:00.09 kjournald
1851 20 0 0 0 0 S 0 0.0 0:00.00 kjournald
1963 20 0 0 0 0 S 0 0.0 0:04.59 flush-8:0
2043 20 0 2008 664 576 S 0 0.1 0:00.23 syslogd
2046 20 0 1892 380 328 S 0 0.0 0:00.04 klogd
2055 20 0 1872 332 236 S 0 0.0 0:00.06 irqbalance
2063 rpc 20 0 2084 500 444 S 0 0.1 0:00.00 portmap
2081 20 0 2116 592 588 S 0 0.1 0:00.05 rpc.statd
2150 20 0 6868 668 576 S 0 0.1 0:00.01 sshd
2198 20 0 6804 444 440 S 0 0.0 0:00.00 sshd
2207 20 0 3280 584 580 S 0 0.1 0:00.01 xinetd
2226 20 0 0 0 0 S 0 0.0 0:00.00 lockd
2227 20 0 0 0 0 S 0 0.0 0:19.59 nfsd
2228 20 0 0 0 0 S 0 0.0 0:25.23 nfsd
2229 20 0 0 0 0 S 0 0.0 0:22.94 nfsd
2230 20 0 0 0 0 S 0 0.0 0:25.19 nfsd
2231 20 0 0 0 0 S 0 0.0 0:16.96 nfsd
2232 20 0 0 0 0 S 0 0.0 0:26.38 nfsd
2233 20 0 0 0 0 S 0 0.0 0:23.47 nfsd
2234 20 0 0 0 0 S 0 0.0 0:18.89 nfsd
2237 20 0 2360 684 552 S 0 0.1 0:00.59 rpc.mountd
2295 0 -20 65244 5096 2052 S 0 0.5 7:31.04 masterd_core
2305 0 -20 27864 1568 1172 S 0 0.2 0:46.44 masterd_manager
2312 15 -5 36460 2164 1304 S 0 0.2 46:17.24 sysd
2314 0 -20 32356 5708 1196 S 0 0.6 10:06.28 masterd_manager
2320 20 0 92040 6656 2580 S 0 0.7 0:02.82 dagger
2321 30 10 40568 3832 1772 S 0 0.4 9:21.69 python
2322 20 0 76640 3024 1532 S 0 0.3 0:01.30 cryptod
2323 20 0 166m 2124 1436 S 0 0.2 1:48.61 sysdagent
2339 20 0 7212 1268 1088 S 0 0.1 0:00.07 tscat
2340 20 0 71580 1240 1080 S 0 0.1 0:07.09 brdagent
2341 20 0 31780 1256 1092 S 0 0.1 0:07.41 ehmon
2342 20 0 47496 1236 1088 S 0 0.1 0:00.57 chasd
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
