topic Re: PA-3050 CPU dataplane issue in General Topics

PA-3050 CPU dataplane issue

TorC — Tue, 17 Jan 2017 14:23:53 GMT

We have a PA-3050 on softwarre 7.0.8 and are struggling with CPU dataplase issues during normal business hours. 80 % load is shown by dashboard in webgui. Session count is approx. 80 000 when issue occurs.

show system resources follow - only says that logrcvr uses 5% and nothing special is elsewhere noticed.

Running command show system statistics session during CPU high load reports approx 500-700 Mbps througput average.

Total supported througput with threat should be 2Gbps and without threat 4 Gbps. Im unsure what happends when threat is enabled, is supported througput then limited to 2 Gbps?

Any advice?

Regards

\\Tor

Re: PA-3050 CPU dataplane issue

BPry — Tue, 17 Jan 2017 14:45:34 GMT

It sounds like that is running a little high with the number of sessions that you have open. That being said it would be interesting to know what your CPS is, if that is reaching near max then these numbers make sense. As far as the logrcvr goes it really depends on how your loggging is setup on your security policies, without looking at more numbers I can't really tell you if that is higher than it should be or not.

Re: PA-3050 CPU dataplane issue

TranceforLife — Tue, 17 Jan 2017 21:32:08 GMT

Hi,

For this sort of issue, l think better to raise a TAC case. They do have a special technics to determine what is actually causing high CPU usage. Or it could be just simple overload or even normal behaviour when handling that much sessions. Did you observe this behaviour all the time on any previous PAN-OS? Could be even a bug in the software:

7.0.10 has a fix but not sure if you facing this bug or not:

Thx,

Myky

Re: PA-3050 CPU dataplane issue

reaper — Wed, 18 Jan 2017 11:02:49 GMT

show system resources is to display Management plane resource usage

show running resource-monitor will give you the dataplane core utilisation:

admin@myNGFW> show running resource-monitor second 

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup                    :     0%
flow_fastpath                  :     0%
flow_slowpath                  :     0%
flow_forwarding                :     0%
flow_mgmt                      :     1%
flow_ctrl                      :     1%
nac_result                     :     0%
flow_np                        :     0%
dfa_result                     :     0%
module_internal                :     0%
aho_result                     :     0%
zip_result                     :     0%
pktlog_forwarding              :     0%
lwm                            :     0%
flow_host                      :     1%

CPU load (%) during last 60 seconds:
core   0   1   2   3   4   5
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0

additionally, show session info will give you an overview of packetrates, bandwidth usage etc:

admin@myNGFW> show session info 

target-dp:                                       *.dp0
--------------------------------------------------------------------------------
Number of sessions supported:                    262142
Number of active sessions:                       2
Number of active TCP sessions:                   0
Number of active UDP sessions:                   2
Number of active ICMP sessions:                  0
Number of active GTPc sessions:                  0
Number of active GTPu sessions:                  0
Number of active BCAST sessions:                 0
Number of active MCAST sessions:                 0
Number of active predict sessions:               0
Session table utilization:                       0%
Number of sessions created since bootup:         25852
Packet rate:                                     32/s
Throughput:                                      15 kbps
New connection establish rate:                   0 cps

When threat is enabled, the average speed we measured in lab was about 2gbps but there is no limitation. some scenarios may have a little less, others will have a little more. it depends on your traffic patern

you could check your hard-software pools to see if any of them are getting depleted

if any of them have a very low number to the left of the / (or even 0), that pool is depleted, which could be a hint at what is causing your issues

admin@myNGFW> debug dataplane pool statistics 


Hardware Pools
[ 0] Packet Buffers            :    11337/11468    0x80000000e7a8f000
[ 1] Work Queue Entries        :   228233/229376   0x80000000e5e8f000
[ 2] Output Buffers            :     4547/4556     0x80000000eea8d000
[ 3] DFA Result                :     3999/4000     0x800000010fa18000
[ 4] Timer Buffers             :     4093/4096     0x800000010f618000
[ 5] LWM Pool                  :     1024/1024     0x800000010f5d8000
[ 6] ZIP Commands              :     1023/1024     0x800000010fe00000
[ 7] Dma Cmd Buffers           :     1016/1024     0x80000000e5d8f000

Software Pools
[ 0] Shared Pool 24            (   24):   557429/560000   0x80000000cc122f80 
[ 1] Shared Pool 32            (   32):   528594/530000   0x80000000cd017280 
[ 2] Shared Pool 40            (   40):    70000/70000    0x80000000ce249700 
[ 3] Shared Pool 192           (  192):  1791975/1795000  0x80000000ce539780 
[ 4] Shared Pool 256           (  256):   139993/140000   0x80000000e34bf180 
[ 5] software packet buffer 0  (  512):    32223/32768    0x8000000095356680 
[ 6] software packet buffer 1  ( 1024):    32768/32768    0x8000000096376780 
[ 7] software packet buffer 2  ( 2048):    81376/81920    0x8000000098396880 
[ 8] software packet buffer 3  (33280):    20480/20480    0x80000000a23e6980 
[ 9] software packet buffer 4  (66048):      304/304      0x80000000cadfaa80 
[10] ZIP Results               (  184):     1024/1024     0x8000000083f9e300 
[11] CTD AV Block              ( 1024):       32/32       0x8000000082bdc380 
[12] Regex Results             (11544):     8000/8000     0x8000000056202880 22
[13] SSH Handshake State       ( 6512):       64/64       0x80000000808e8680 
[14] SSH State                 ( 3200):      512/512      0x800000008094e480 
[15] TCP host connections      (  176):       15/16       0x8000000080adf180

Re: PA-3050 CPU dataplane issue

TranceforLife — Wed, 18 Jan 2017 11:08:09 GMT

Add to the comment above this KB might help:

https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/FeaturedArticles/article-id/147

Re: PA-3050 CPU dataplane issue

TorC — Thu, 19 Jan 2017 12:20:32 GMT

Thanks everyone for good advices. The setup we have is actualle an active/Passive HA pair and we did yesterday an upgrade from 7.0.8 to 7.0.13 to see it that helped. Unfortunately not:-) I checked the hardware pools and they wre ok.

When you enable threat are you then limited to 2 gbps throughput?

Here are statistics when load is 75% approx. Cant find anything peculiar.

show running resource-monitor hour

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       0   0 55 83 65 89 66 89 67 90 69 90
       0   0 60 83 70 90 70 90 71 90 72 91
       0   0 61 85 72 92 73 92 74 92 75 92
       0   0 52 81 65 88 66 88 67 89 68 89
       0   0 20 70 30 71 31 69 33 67 36 68
       0   0   9 19 13 29 14 32 15 34 18 38
       0   0   8 54 10 86 11 86 13 87 15 87
       0   0   8 14 11 26 12 27 13 30 15 33
       0   0   9 15 11 31 12 32 13 33 15 35
       0   0   9 15 11 24 12 25 13 27 16 30
       0   0   9 19 11 28 12 29 13 31 15 33
       0   0   9 21 11 27 12 28 14 29 16 31
       0   0 10 20 12 24 13 24 15 25 17 28
       0   0 10 19 13 35 14 36 15 38 18 40
       0   0 11 44 14 39 15 40 17 40 19 42
       0   0 11 23 13 29 15 31 16 31 19 33
       0   0 12 23 15 29 17 30 18 36 21 39
       0   0 13 25 15 31 17 32 18 34 21 36
       0   0 12 30 15 40 17 38 18 38 21 40
       0   0 11 26 14 44 16 46 17 48 20 50
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0

Resource utilization (%) during last 24 hours:
session (average):
15 16 16 15   7   4   3   3   3   3   3   4   4   4   4
4   4   4   5   5   0   0   0   0
session (maximum):
17 17 19 20 11   4   5   4   4   4   4   4   4   5   5
5   5   5   5   7   0   0   0   0
packet buffer (average):
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0
packet buffer (maximum):
2   2   3   3   1   1   7   1   1   0   0   0   1   1   2
1   2   2   1   2   0   0   0   0
packet descriptor (average):
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0
packet descriptor (maximum):
1   1   1   1   0   0   3   0   0   0   0   0   1   0   1
0   1   1   0   1   0   0   0   0
packet descriptor (on-chip) (average):
3   3   3   3   2   2   2   2   2   2   2   2   2   2   2
2   2   2   2   2   0   0   0   0
packet descriptor (on-chip) (maximum):
38 37 43 44 23 16 12 10 12   5 11   4 16 11 11
20 13 18 30 23   0   0   0   0

HA1(active)>

Re: PA-3050 CPU dataplane issue

MohamedSharief — Sun, 05 Feb 2017 06:36:10 GMT

Hi TorC,

Consider using App override policy for sources/destinations that are mostly consuming your bandwidth.

You can use below KB it might help you:

https://live.paloaltonetworks.com/t5/Learning-Articles/Improving-Performance-of-HTTP-with-DSRI/ta-p/56411

Regards,

Sharief