topic Re: PA-500 CLI and web interface unbearably slow in General Topics

PA-500 CLI and web interface unbearably slow

cstech — Thu, 21 Aug 2014 23:03:46 GMT

"show system resources" indicates CPU is 80% idle... yet can take 90 seconds to display results. Logging into the web interface can take 60 seconds. Committing insignificant changes can take 5 minutes (Maybe this is excusable). These are not exaggerations.

How can I troubleshoot the source of this latency? I'd love to hear an responses such as "you don't have logging turned on, do you? go turn it off!" or "get used to it, we all have that problem".

I'm running 6.0.2.

Thank you,

Chris

Re: PA-500 CLI and web interface unbearably slow

HULK — Fri, 22 Aug 2014 00:24:49 GMT

Hello Chris,

Could you please apply CLI command > show system resources follow ( press enter). After that, apply Shift +M. Capture that output and share with us. It may be web-server or management-server is taking high memory on the MP.

Thanks

Re: PA-500 CLI and web interface unbearably slow

Strachf — Fri, 22 Aug 2014 03:03:22 GMT

I am getting the same issue on my PA-2050 using PA OS 6.0.3

top - 12:38:40 up 46 days, 4:06, 1 user, load average: 16.20, 14.47, 9.41

Mem: 942232k total, 918288k used, 23944k free, 872k buffers

Swap: 2008084k total, 1206204k used, 801880k free, 97004k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2540 root 20 0 946m 287m 4188 S 16 31.2 1326:32 mgmtsrvr

2539 root 20 0 364m 89m 7904 S 0 9.7 1736:31 devsrvr

2564 root 20 0 691m 74m 1696 S 0 8.1 540:43.88 logrcvr

2523 root 20 0 195m 67m 63m S 0 7.4 537:13.90 useridd

27243 nobody 20 0 230m 30m 4320 S 0 3.3 97:26.05 appweb3

25316 root 30 10 61384 14m 5444 D 0 1.6 0:00.91 pan_logdb_index

14141 nobody 20 0 203m 14m 3296 S 65 1.6 1167:41 appweb3

2087 root 0 -20 79320 8940 2068 S 0 0.9 389:30.83 masterd_apps

2570 root 20 0 207m 5660 2332 S 0 0.6 92:02.36 sslmgr

13805 root 20 0 5376 5252 2748 S 0 0.6 0:01.56 ntpd

21338 admin 20 0 106m 4336 832 S 0 0.5 0:06.85 cli

2113 root 20 0 99.7m 4060 1280 S 0 0.4 19:28.60 dagger

2114 root 30 10 42836 3848 1740 S 0 0.4 220:48.80 python

2575 root 20 0 116m 3304 1892 S 0 0.4 87:05.35 authd

2098 root 15 -5 41332 2348 1216 S 0 0.2 723:19.93 sysd

2572 root 20 0 82668 1976 1180 S 0 0.2 31:04.69 dnsproxyd

2574 root 20 0 147m 1876 1152 S 0 0.2 34:44.52 routed

2567 root 20 0 217m 1788 1424 S 0 0.2 111:03.09 varrcvr

2121 root 20 0 137m 1712 1192 S 0 0.2 43:44.09 sysdagent

14106 nobody 20 0 104m 1664 1372 S 0 0.2 17:01.23 appweb3

2350 root 20 0 95244 1576 1208 S 0 0.2 22:37.49 cryptod

2569 root 20 0 116m 1476 1228 S 0 0.2 29:17.18 satd

2563 root 20 0 84236 1420 1228 S 0 0.2 21:56.60 ikemgr

2145 root 20 0 84956 1380 1076 S 0 0.1 30:29.08 chasd

2568 root 17 -3 66604 1376 1052 S 0 0.1 26:05.06 ha_agent

2573 root 20 0 80340 1248 1092 S 0 0.1 23:00.67 pppoed

2565 root 20 0 102m 1236 1064 S 0 0.1 23:11.16 rasmgr

5058 root 20 0 49888 1236 1032 S 0 0.1 32:18.81 snmpd

2571 root 20 0 67204 1188 1056 S 0 0.1 26:40.90 pan_dhcpd

25331 root 20 0 3828 1172 1036 D 0 0.1 0:00.23 sh

2144 root 20 0 36052 1128 972 S 0 0.1 40:15.44 ehmon

2566 root 20 0 101m 1124 992 S 0 0.1 23:35.00 keymgr

2143 root 20 0 67660 1080 948 S 0 0.1 25:17.97 brdagent

21305 root 20 0 21316 1080 1016 S 0 0.1 0:00.30 sshd

25228 admin 20 0 4528 1060 808 R 49 0.1 0:02.00 top

21780 root 30 10 3892 1052 880 S 0 0.1 0:00.79 genindex.sh

21333 admin 20 0 21452 820 640 S 0 0.1 0:00.40 sshd

5063 root 20 0 8792 788 680 S 0 0.1 0:18.63 syslog-ng

2391 root 20 0 6844 780 752 S 0 0.1 0:00.11 sshd

1912 root 20 0 2108 668 664 S 0 0.1 0:00.06 rpc.statd

2392 root 20 0 6780 668 664 S 0 0.1 0:00.05 sshd

13778 root 20 0 2896 656 596 S 0 0.1 0:00.18 crond

1874 root 20 0 2004 632 588 S 0 0.1 0:14.43 syslogd

1977 root 20 0 3276 628 624 S 0 0.1 0:00.02 xinetd

2007 root 20 0 2608 628 560 S 0 0.1 0:17.35 rpc.mountd

2142 root 20 0 7208 624 620 S 0 0.1 0:00.13 tscat

1 root 20 0 1832 600 576 S 0 0.1 1:10.71 init

1894 rpc 20 0 2076 484 480 S 0 0.1 0:00.01 portmap

2090 root 20 0 1884 456 452 S 0 0.0 0:00.01 agetty

867 root 16 -4 1992 384 380 S 0 0.0 0:01.06 udevd

1877 root 20 0 1888 368 364 S 0 0.0 0:00.05 klogd

1886 root 20 0 1868 328 232 S 0 0.0 13:44.25 irqbalance

5062 root 20 0 4796 212 208 S 0 0.0 0:00.01 syslog-ng

2 root 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd

3 root RT 0 0 0 0 S 0 0.0 0:24.12 migration/0

4 root 20 0 0 0 0 S 0 0.0 8:24.78 ksoftirqd/0

5 root RT 0 0 0 0 S 0 0.0 0:23.39 migration/1

6 root 20 0 0 0 0 S 0 0.0 8:04.71 ksoftirqd/1

Re: PA-500 CLI and web interface unbearably slow

HULK — Fri, 22 Aug 2014 03:46:24 GMT

Hello Chris,

Could you please restart the management server daemon on this PAN FW. It seems the management-server takes a large amount of memory from it's MP. This will not impact to the production traffic on your PAN firewall.

CLI command to restart mgmt-server: > debug software restart management-server

( after applying the above mention command, you will lose the SSH access to the fw)

Mem: 942232k total >>>>>>> total memory available on Management-plane, 918288k used, 23944k free,

2540 root 20 0 946m >>>>>>>>>>>>>> memory used 287m 4188 S 16 31.2 1326:32 mgmtsrvr

Thanks

Re: PA-500 CLI and web interface unbearably slow

cstech — Fri, 22 Aug 2014 16:41:42 GMT

Hulk,

Thank you. I should have thought to look at mem usage after I confirmed it wasn't a CPU problem. mgmtsrvr mem usage dropped from 1GB to 400MB. Now I have to figure out how to setup a cron job to restart it every night (only partially joking).

Chris

Re: PA-500 CLI and web interface unbearably slow

HULK — Fri, 22 Aug 2014 16:50:13 GMT

You are welcome Chris.

Re: PA-500 CLI and web interface unbearably slow

licenselu — Fri, 22 Aug 2014 18:51:09 GMT

Hello,

If I were you, I will upgrade the memory (upgrade kit exist for about 500$, with the upgrade you got 2GB for the management plane).

Navigation into the web interface is MUCH more faster and commit time is divided by a factor of 2...

Regards,

Re: PA-500 CLI and web interface unbearably slow

diennea — Fri, 29 Aug 2014 11:07:42 GMT

Hi,I've got the same problem with PA500 and PAN OS 6.0.4

In my case the process geindex.sh use a lot of cpu

ORDER BY CPU

top - 13:04:59 up 10 days, 18:26, 1 user, load average: 1.24, 1.58, 3.50
Tasks: 104 total,   3 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 3.7%us, 5.8%sy, 11.2%ni, 68.2%id, 10.9%wa, 0.0%hi, 0.2%si, 0.0%st
Mem:    938136k total,   897268k used,    40868k free,    34108k buffers
Swap: 2008084k total, 1521284k used,   486800k free,   111064k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
22839       30 10 4012 1384 1088 R   60 0.1   1:33.20 genindex.sh
7235       20   0 4464 1016 784 R    4 0.1   0:00.05 top
2972       20   0 102m 1448 1072 S    2 0.2   6:26.11 keymgr
20881       20   0 49912 2100 1388 S    2 0.2   8:28.13 snmpd
    1       20   0 1832 544 516 S    0 0.1   0:15.38 init
    2       20   0     0    0    0 S    0 0.0   0:00.01 kthreadd
    3       RT   0     0    0    0 S    0 0.0   0:03.61 migration/0
    4       20   0     0    0    0 S    0 0.0   1:25.80 ksoftirqd/0
    5       RT   0     0    0    0 S    0 0.0   0:03.97 migration/1
    6       20   0     0    0    0 S    0 0.0   1:21.47 ksoftirqd/1
    7       20   0     0    0    0 S    0 0.0   2:43.78 events/0
    8       20   0     0    0    0 S    0 0.0   4:18.32 events/1
    9       20   0     0    0    0 S    0 0.0   0:00.05 khelper
   12       20   0     0    0    0 S    0 0.0   0:00.00 async/mgr
111       20   0     0    0    0 S    0 0.0   0:01.70 sync_supers
113       20   0     0    0    0 S    0 0.0   0:02.73 bdi-default
114       20   0     0    0    0 S    0 0.0   1:19.88 kblockd/0
115       20   0     0    0    0 S    0 0.0   1:16.84 kblockd/1
124       20   0     0    0    0 S    0 0.0   0:00.00 ata/0
125       20   0     0    0    0 S    0 0.0   0:00.00 ata/1
126       20   0     0    0    0 S    0 0.0   0:00.00 ata_aux
131       20   0     0    0    0 S    0 0.0   0:00.00 khubd
134       20   0     0    0    0 S    0 0.0   0:00.00 kseriod
155       20   0     0    0    0 S    0 0.0   0:00.00 rpciod/0
156       20   0     0    0    0 S    0 0.0   0:00.00 rpciod/1
172       20   0     0    0    0 R    0 0.0 27:54.79 kswapd0
173       20   0     0    0    0 S    0 0.0   0:00.00 aio/0
174       20   0     0    0    0 S    0 0.0   0:00.00 aio/1
175        1 -19     0    0    0 S    0 0.0   0:00.00 nfsiod
765       20   0     0    0    0 S    0 0.0   1:44.30 octeon-ethernet
793       20   0     0    0    0 S    0 0.0   0:00.00 scsi_eh_0
798       20   0     0    0    0 S    0 0.0   0:01.18 mtdblockd
826       20   0     0    0    0 S    0 0.0   0:00.00 usbhid_resumer
866       20   0     0    0    0 S    0 0.0   0:43.94 kjournald
919       16 -4 1992 392 388 S    0 0.0   0:01.03 udevd
1172       20   0 21316 2468 2036 S    0 0.3   0:00.14 sshd

ORDER BY MEMORY

top - 13:06:19 up 10 days, 18:27, 1 user, load average: 0.91, 1.42, 3.28
Tasks: 104 total,   3 running, 101 sleeping,   0 stopped,   0 zombie
Cpu(s): 0.8%us, 10.5%sy, 27.2%ni, 30.7%id, 30.7%wa, 0.2%hi, 0.0%si, 0.0%st
Mem:    938136k total,   900764k used,    37372k free,    30892k buffers
Swap: 2008084k total, 1521292k used,   486792k free,   109728k cached

PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+ COMMAND
5843 root      20   0 863m 86m 2824 S    0 9.5 98:37.92 mgmtsrvr
2970 root      20   0 1074m 85m 1664 D    1 9.4 44:07.18 logrcvr
2945 root      20   0 242m 83m 8664 S    0 9.1 111:50.75 devsrvr
2934 root      20   0 183m 69m 63m S    0 7.6 64:02.18 useridd
1818 mauro.bi 20   0 106m 25m 11m S    0 2.8   0:03.50 cli
7432 nobody    20   0 271m 15m 1552 S    0 1.7 86:48.31 appweb3
2290 root       0 -20 85888 8292 2000 S    0 0.9 65:17.81 masterd_apps
8505 nobody    20   0 111m 6112 3180 S    0 0.7   7:58.57 appweb3
22664 root      20   0 37276 6044 3324 S    0 0.6   0:00.79 masterd_batch
8503 nobody    20   0 176m 5500 2504 S    0 0.6   8:07.93 appweb3
20681 root      20   0 5616 5480 2952 S    0 0.6   0:00.59 ntpd
2981 root      20   0 107m 4320 3632 S    0 0.5   4:02.39 authd
2318 root      30 10 42968 3732 1668 S    0 0.4 40:58.18 python
2978 root      20   0 82772 3640 1120 S    0 0.4   5:10.94 dnsproxyd
1172 root      20   0 21316 2416 1984 S    0 0.3   0:00.14 sshd
2302 root      15 -5 41216 2268 1200 S    0 0.2 117:49.65 sysd
2969 root      20   0 85144 2248 1688 S    0 0.2   5:57.16 ikemgr
2980 root      20   0 147m 2104 1128 S    0 0.2   6:19.41 routed
20881 root      20   0 49912 2092 1384 S    0 0.2   8:28.39 snmpd
2973 root      20   0 217m 1972 1292 S    0 0.2 38:48.62 varrcvr
2325 root      20   0 139m 1940 1164 S    0 0.2   7:22.91 sysdagent
2971 root      20   0 103m 1892 1460 S    0 0.2   4:32.53 rasmgr
2317 root      20   0 100m 1716 1268 S    0 0.2   3:45.53 dagger
2616 root      20   0 95912 1684 1188 S    0 0.2   4:17.02 cryptod
10375 root      20   0 12124 1668 1508 S    0 0.2   0:00.02 wmic

Any idea?

Thanks

Regards

Re: PA-500 CLI and web interface unbearably slow

ajbool — Fri, 29 Aug 2014 11:29:29 GMT

Vendor RAM pricing; don't you just love it 😉

Re: PA-500 CLI and web interface unbearably slow

lios — Fri, 03 Oct 2014 11:14:35 GMT

I can just confirm that PA-500 management (PanOS 6.0.5 now, the same with earlier versions) is EXTREMELY slow. Actually, it's a shame that PA's software can not normally run on it's hardware!

Re: PA-500 CLI and web interface unbearably slow

diennea — Fri, 03 Oct 2014 11:53:41 GMT

I suggest you to buy RAM upgrade.

It solves my problems

Now GUI is pretty fast and commit operation take less than 3 minutes

Re: PA-500 CLI and web interface unbearably slow

lios — Fri, 03 Oct 2014 12:15:19 GMT

I think that 3 minutes to commit is still slow.. How much time booting takes? Ours (without ram upgrade) takes more than 10min to boot and is pain to manage.. PA is selling expensive RAM upgrade kits instead of apologizing it's customers and giving them for free..

Actually, we are looking into other vendors.