08-21-2014 04:03 PM
"show system resources" indicates CPU is 80% idle... yet can take 90 seconds to display results. Logging into the web interface can take 60 seconds. Committing insignificant changes can take 5 minutes (Maybe this is excusable). These are not exaggerations.
How can I troubleshoot the source of this latency? I'd love to hear an responses such as "you don't have logging turned on, do you? go turn it off!" or "get used to it, we all have that problem".
I'm running 6.0.2.
Thank you,
Chris
08-21-2014 08:46 PM
Hello Chris,
Could you please restart the management server daemon on this PAN FW. It seems the management-server takes a large amount of memory from it's MP. This will not impact to the production traffic on your PAN firewall.
CLI command to restart mgmt-server: > debug software restart management-server
( after applying the above mention command, you will lose the SSH access to the fw)
Mem: 942232k total >>>>>>> total memory available on Management-plane, 918288k used, 23944k free,
2540 root 20 0 946m >>>>>>>>>>>>>> memory used 287m 4188 S 16 31.2 1326:32 mgmtsrvr
Thanks
08-21-2014 05:24 PM
Hello Chris,
Could you please apply CLI command > show system resources follow ( press enter). After that, apply Shift +M. Capture that output and share with us. It may be web-server or management-server is taking high memory on the MP.
Thanks
08-21-2014 08:03 PM
I am getting the same issue on my PA-2050 using PA OS 6.0.3
top - 12:38:40 up 46 days, 4:06, 1 user, load average: 16.20, 14.47, 9.41
Mem: 942232k total, 918288k used, 23944k free, 872k buffers
Swap: 2008084k total, 1206204k used, 801880k free, 97004k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2540 root 20 0 946m 287m 4188 S 16 31.2 1326:32 mgmtsrvr
2539 root 20 0 364m 89m 7904 S 0 9.7 1736:31 devsrvr
2564 root 20 0 691m 74m 1696 S 0 8.1 540:43.88 logrcvr
2523 root 20 0 195m 67m 63m S 0 7.4 537:13.90 useridd
27243 nobody 20 0 230m 30m 4320 S 0 3.3 97:26.05 appweb3
25316 root 30 10 61384 14m 5444 D 0 1.6 0:00.91 pan_logdb_index
14141 nobody 20 0 203m 14m 3296 S 65 1.6 1167:41 appweb3
2087 root 0 -20 79320 8940 2068 S 0 0.9 389:30.83 masterd_apps
2570 root 20 0 207m 5660 2332 S 0 0.6 92:02.36 sslmgr
13805 root 20 0 5376 5252 2748 S 0 0.6 0:01.56 ntpd
21338 admin 20 0 106m 4336 832 S 0 0.5 0:06.85 cli
2113 root 20 0 99.7m 4060 1280 S 0 0.4 19:28.60 dagger
2114 root 30 10 42836 3848 1740 S 0 0.4 220:48.80 python
2575 root 20 0 116m 3304 1892 S 0 0.4 87:05.35 authd
2098 root 15 -5 41332 2348 1216 S 0 0.2 723:19.93 sysd
2572 root 20 0 82668 1976 1180 S 0 0.2 31:04.69 dnsproxyd
2574 root 20 0 147m 1876 1152 S 0 0.2 34:44.52 routed
2567 root 20 0 217m 1788 1424 S 0 0.2 111:03.09 varrcvr
2121 root 20 0 137m 1712 1192 S 0 0.2 43:44.09 sysdagent
14106 nobody 20 0 104m 1664 1372 S 0 0.2 17:01.23 appweb3
2350 root 20 0 95244 1576 1208 S 0 0.2 22:37.49 cryptod
2569 root 20 0 116m 1476 1228 S 0 0.2 29:17.18 satd
2563 root 20 0 84236 1420 1228 S 0 0.2 21:56.60 ikemgr
2145 root 20 0 84956 1380 1076 S 0 0.1 30:29.08 chasd
2568 root 17 -3 66604 1376 1052 S 0 0.1 26:05.06 ha_agent
2573 root 20 0 80340 1248 1092 S 0 0.1 23:00.67 pppoed
2565 root 20 0 102m 1236 1064 S 0 0.1 23:11.16 rasmgr
5058 root 20 0 49888 1236 1032 S 0 0.1 32:18.81 snmpd
2571 root 20 0 67204 1188 1056 S 0 0.1 26:40.90 pan_dhcpd
25331 root 20 0 3828 1172 1036 D 0 0.1 0:00.23 sh
2144 root 20 0 36052 1128 972 S 0 0.1 40:15.44 ehmon
2566 root 20 0 101m 1124 992 S 0 0.1 23:35.00 keymgr
2143 root 20 0 67660 1080 948 S 0 0.1 25:17.97 brdagent
21305 root 20 0 21316 1080 1016 S 0 0.1 0:00.30 sshd
25228 admin 20 0 4528 1060 808 R 49 0.1 0:02.00 top
21780 root 30 10 3892 1052 880 S 0 0.1 0:00.79 genindex.sh
21333 admin 20 0 21452 820 640 S 0 0.1 0:00.40 sshd
5063 root 20 0 8792 788 680 S 0 0.1 0:18.63 syslog-ng
2391 root 20 0 6844 780 752 S 0 0.1 0:00.11 sshd
1912 root 20 0 2108 668 664 S 0 0.1 0:00.06 rpc.statd
2392 root 20 0 6780 668 664 S 0 0.1 0:00.05 sshd
13778 root 20 0 2896 656 596 S 0 0.1 0:00.18 crond
1874 root 20 0 2004 632 588 S 0 0.1 0:14.43 syslogd
1977 root 20 0 3276 628 624 S 0 0.1 0:00.02 xinetd
2007 root 20 0 2608 628 560 S 0 0.1 0:17.35 rpc.mountd
2142 root 20 0 7208 624 620 S 0 0.1 0:00.13 tscat
1 root 20 0 1832 600 576 S 0 0.1 1:10.71 init
1894 rpc 20 0 2076 484 480 S 0 0.1 0:00.01 portmap
2090 root 20 0 1884 456 452 S 0 0.0 0:00.01 agetty
867 root 16 -4 1992 384 380 S 0 0.0 0:01.06 udevd
1877 root 20 0 1888 368 364 S 0 0.0 0:00.05 klogd
1886 root 20 0 1868 328 232 S 0 0.0 13:44.25 irqbalance
5062 root 20 0 4796 212 208 S 0 0.0 0:00.01 syslog-ng
2 root 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0 0.0 0:24.12 migration/0
4 root 20 0 0 0 0 S 0 0.0 8:24.78 ksoftirqd/0
5 root RT 0 0 0 0 S 0 0.0 0:23.39 migration/1
6 root 20 0 0 0 0 S 0 0.0 8:04.71 ksoftirqd/1
08-21-2014 08:46 PM
Hello Chris,
Could you please restart the management server daemon on this PAN FW. It seems the management-server takes a large amount of memory from it's MP. This will not impact to the production traffic on your PAN firewall.
CLI command to restart mgmt-server: > debug software restart management-server
( after applying the above mention command, you will lose the SSH access to the fw)
Mem: 942232k total >>>>>>> total memory available on Management-plane, 918288k used, 23944k free,
2540 root 20 0 946m >>>>>>>>>>>>>> memory used 287m 4188 S 16 31.2 1326:32 mgmtsrvr
Thanks
08-22-2014 09:41 AM
Hulk,
Thank you. I should have thought to look at mem usage after I confirmed it wasn't a CPU problem. mgmtsrvr mem usage dropped from 1GB to 400MB. Now I have to figure out how to setup a cron job to restart it every night (only partially joking).
Chris
08-22-2014 09:50 AM
You are welcome Chris. 
08-22-2014 11:51 AM
Hello,
If I were you, I will upgrade the memory (upgrade kit exist for about 500$, with the upgrade you got 2GB for the management plane).
Navigation into the web interface is MUCH more faster and commit time is divided by a factor of 2...
Regards,
HA
08-29-2014 04:07 AM
Hi,I've got the same problem with PA500 and PAN OS 6.0.4
In my case the process geindex.sh use a lot of cpu
ORDER BY CPU
top - 13:04:59 up 10 days, 18:26, 1 user, load average: 1.24, 1.58, 3.50
Tasks: 104 total, 3 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 5.8%sy, 11.2%ni, 68.2%id, 10.9%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 938136k total, 897268k used, 40868k free, 34108k buffers
Swap: 2008084k total, 1521284k used, 486800k free, 111064k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22839 30 10 4012 1384 1088 R 60 0.1 1:33.20 genindex.sh
7235 20 0 4464 1016 784 R 4 0.1 0:00.05 top
2972 20 0 102m 1448 1072 S 2 0.2 6:26.11 keymgr
20881 20 0 49912 2100 1388 S 2 0.2 8:28.13 snmpd
1 20 0 1832 544 516 S 0 0.1 0:15.38 init
2 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd
3 RT 0 0 0 0 S 0 0.0 0:03.61 migration/0
4 20 0 0 0 0 S 0 0.0 1:25.80 ksoftirqd/0
5 RT 0 0 0 0 S 0 0.0 0:03.97 migration/1
6 20 0 0 0 0 S 0 0.0 1:21.47 ksoftirqd/1
7 20 0 0 0 0 S 0 0.0 2:43.78 events/0
8 20 0 0 0 0 S 0 0.0 4:18.32 events/1
9 20 0 0 0 0 S 0 0.0 0:00.05 khelper
12 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr
111 20 0 0 0 0 S 0 0.0 0:01.70 sync_supers
113 20 0 0 0 0 S 0 0.0 0:02.73 bdi-default
114 20 0 0 0 0 S 0 0.0 1:19.88 kblockd/0
115 20 0 0 0 0 S 0 0.0 1:16.84 kblockd/1
124 20 0 0 0 0 S 0 0.0 0:00.00 ata/0
125 20 0 0 0 0 S 0 0.0 0:00.00 ata/1
126 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux
131 20 0 0 0 0 S 0 0.0 0:00.00 khubd
134 20 0 0 0 0 S 0 0.0 0:00.00 kseriod
155 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/0
156 20 0 0 0 0 S 0 0.0 0:00.00 rpciod/1
172 20 0 0 0 0 R 0 0.0 27:54.79 kswapd0
173 20 0 0 0 0 S 0 0.0 0:00.00 aio/0
174 20 0 0 0 0 S 0 0.0 0:00.00 aio/1
175 1 -19 0 0 0 S 0 0.0 0:00.00 nfsiod
765 20 0 0 0 0 S 0 0.0 1:44.30 octeon-ethernet
793 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
798 20 0 0 0 0 S 0 0.0 0:01.18 mtdblockd
826 20 0 0 0 0 S 0 0.0 0:00.00 usbhid_resumer
866 20 0 0 0 0 S 0 0.0 0:43.94 kjournald
919 16 -4 1992 392 388 S 0 0.0 0:01.03 udevd
1172 20 0 21316 2468 2036 S 0 0.3 0:00.14 sshd
ORDER BY MEMORY
top - 13:06:19 up 10 days, 18:27, 1 user, load average: 0.91, 1.42, 3.28
Tasks: 104 total, 3 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 10.5%sy, 27.2%ni, 30.7%id, 30.7%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 938136k total, 900764k used, 37372k free, 30892k buffers
Swap: 2008084k total, 1521292k used, 486792k free, 109728k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5843 root 20 0 863m 86m 2824 S 0 9.5 98:37.92 mgmtsrvr
2970 root 20 0 1074m 85m 1664 D 1 9.4 44:07.18 logrcvr
2945 root 20 0 242m 83m 8664 S 0 9.1 111:50.75 devsrvr
2934 root 20 0 183m 69m 63m S 0 7.6 64:02.18 useridd
1818 mauro.bi 20 0 106m 25m 11m S 0 2.8 0:03.50 cli
7432 nobody 20 0 271m 15m 1552 S 0 1.7 86:48.31 appweb3
2290 root 0 -20 85888 8292 2000 S 0 0.9 65:17.81 masterd_apps
8505 nobody 20 0 111m 6112 3180 S 0 0.7 7:58.57 appweb3
22664 root 20 0 37276 6044 3324 S 0 0.6 0:00.79 masterd_batch
8503 nobody 20 0 176m 5500 2504 S 0 0.6 8:07.93 appweb3
20681 root 20 0 5616 5480 2952 S 0 0.6 0:00.59 ntpd
2981 root 20 0 107m 4320 3632 S 0 0.5 4:02.39 authd
2318 root 30 10 42968 3732 1668 S 0 0.4 40:58.18 python
2978 root 20 0 82772 3640 1120 S 0 0.4 5:10.94 dnsproxyd
1172 root 20 0 21316 2416 1984 S 0 0.3 0:00.14 sshd
2302 root 15 -5 41216 2268 1200 S 0 0.2 117:49.65 sysd
2969 root 20 0 85144 2248 1688 S 0 0.2 5:57.16 ikemgr
2980 root 20 0 147m 2104 1128 S 0 0.2 6:19.41 routed
20881 root 20 0 49912 2092 1384 S 0 0.2 8:28.39 snmpd
2973 root 20 0 217m 1972 1292 S 0 0.2 38:48.62 varrcvr
2325 root 20 0 139m 1940 1164 S 0 0.2 7:22.91 sysdagent
2971 root 20 0 103m 1892 1460 S 0 0.2 4:32.53 rasmgr
2317 root 20 0 100m 1716 1268 S 0 0.2 3:45.53 dagger
2616 root 20 0 95912 1684 1188 S 0 0.2 4:17.02 cryptod
10375 root 20 0 12124 1668 1508 S 0 0.2 0:00.02 wmic
Any idea?
Thanks
Regards
08-29-2014 04:29 AM
Vendor RAM pricing; don't you just love it 😉
10-03-2014 04:14 AM
I can just confirm that PA-500 management (PanOS 6.0.5 now, the same with earlier versions) is EXTREMELY slow. Actually, it's a shame that PA's software can not normally run on it's hardware!
10-03-2014 04:53 AM
I suggest you to buy RAM upgrade.
It solves my problems
Now GUI is pretty fast and commit operation take less than 3 minutes
10-03-2014 05:15 AM
I think that 3 minutes to commit is still slow.. How much time booting takes? Ours (without ram upgrade) takes more than 10min to boot and is pain to manage.. 
PA is selling expensive RAM upgrade kits instead of apologizing it's customers and giving them for free..
Actually, we are looking into other vendors.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
