PA-3050 CPU dataplane issue

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

PA-3050 CPU dataplane issue

L1 Bithead

We have a PA-3050 on softwarre 7.0.8 and are struggling with CPU dataplase issues during normal business hours.   80 % load is shown by dashboard in webgui.  Session count is approx. 80 000 when issue occurs.

 

 show system resources follow  - only says that logrcvr uses 5% and nothing special is elsewhere noticed.

Running command show system statistics session during CPU high load reports approx 500-700 Mbps througput average. 

 

Total supported througput with threat should be 2Gbps and without threat 4 Gbps. Im unsure what happends when threat is enabled, is supported througput then limited to 2 Gbps?

 

Any advice?

 


Regards

 

\\Tor

6 REPLIES 6

Cyber Elite
Cyber Elite

It sounds like that is running a little high with the number of sessions that you have open. That being said it would be interesting to know what your CPS is, if that is reaching near max then these numbers make sense. As far as the logrcvr goes it really depends on how your loggging is setup on your security policies, without looking at more numbers I can't really tell you if that is higher than it should be or not. 

L6 Presenter

Hi,

 

For this sort of issue, l think better to raise a TAC case. They do have a special technics to determine what is actually causing high CPU usage. Or it could be just simple overload or even normal behaviour when handling that much sessions. Did you observe this behaviour all the time on any previous PAN-OS? Could be even a bug in the software:

 

7.0.10 has a fix but not sure if you facing this bug or not:

 

CPU.PNG

 

Thx,

Myky

Cyber Elite
Cyber Elite

show system resources is to display Management plane resource usage

show running resource-monitor will give you the dataplane core utilisation:

 

admin@myNGFW> show running resource-monitor second 

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup                    :     0%
flow_fastpath                  :     0%
flow_slowpath                  :     0%
flow_forwarding                :     0%
flow_mgmt                      :     1%
flow_ctrl                      :     1%
nac_result                     :     0%
flow_np                        :     0%
dfa_result                     :     0%
module_internal                :     0%
aho_result                     :     0%
zip_result                     :     0%
pktlog_forwarding              :     0%
lwm                            :     0%
flow_host                      :     1%

CPU load (%) during last 60 seconds:
core   0   1   2   3   4   5
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0
       *   1   0   0   0   0

additionally, show session info will give you an overview of packetrates, bandwidth usage etc:

admin@myNGFW> show session info 

target-dp:                                       *.dp0
--------------------------------------------------------------------------------
Number of sessions supported:                    262142
Number of active sessions:                       2
Number of active TCP sessions:                   0
Number of active UDP sessions:                   2
Number of active ICMP sessions:                  0
Number of active GTPc sessions:                  0
Number of active GTPu sessions:                  0
Number of active BCAST sessions:                 0
Number of active MCAST sessions:                 0
Number of active predict sessions:               0
Session table utilization:                       0%
Number of sessions created since bootup:         25852
Packet rate:                                     32/s
Throughput:                                      15 kbps
New connection establish rate:                   0 cps

 

When threat is enabled, the average speed we measured in lab was about 2gbps but there is no limitation. some scenarios may have a little less, others will have a little more. it depends on your  traffic patern

 

you could check your hard-software pools to see if any of them are getting depleted

if any of them have a very low number to the left of the / (or even 0), that pool is depleted, which could be a hint at what is causing your issues

 

admin@myNGFW> debug dataplane pool statistics 


Hardware Pools
[ 0] Packet Buffers            :    11337/11468    0x80000000e7a8f000
[ 1] Work Queue Entries        :   228233/229376   0x80000000e5e8f000
[ 2] Output Buffers            :     4547/4556     0x80000000eea8d000
[ 3] DFA Result                :     3999/4000     0x800000010fa18000
[ 4] Timer Buffers             :     4093/4096     0x800000010f618000
[ 5] LWM Pool                  :     1024/1024     0x800000010f5d8000
[ 6] ZIP Commands              :     1023/1024     0x800000010fe00000
[ 7] Dma Cmd Buffers           :     1016/1024     0x80000000e5d8f000

Software Pools
[ 0] Shared Pool 24            (   24):   557429/560000   0x80000000cc122f80 
[ 1] Shared Pool 32            (   32):   528594/530000   0x80000000cd017280 
[ 2] Shared Pool 40            (   40):    70000/70000    0x80000000ce249700 
[ 3] Shared Pool 192           (  192):  1791975/1795000  0x80000000ce539780 
[ 4] Shared Pool 256           (  256):   139993/140000   0x80000000e34bf180 
[ 5] software packet buffer 0  (  512):    32223/32768    0x8000000095356680 
[ 6] software packet buffer 1  ( 1024):    32768/32768    0x8000000096376780 
[ 7] software packet buffer 2  ( 2048):    81376/81920    0x8000000098396880 
[ 8] software packet buffer 3  (33280):    20480/20480    0x80000000a23e6980 
[ 9] software packet buffer 4  (66048):      304/304      0x80000000cadfaa80 
[10] ZIP Results               (  184):     1024/1024     0x8000000083f9e300 
[11] CTD AV Block              ( 1024):       32/32       0x8000000082bdc380 
[12] Regex Results             (11544):     8000/8000     0x8000000056202880 22
[13] SSH Handshake State       ( 6512):       64/64       0x80000000808e8680 
[14] SSH State                 ( 3200):      512/512      0x800000008094e480 
[15] TCP host connections      (  176):       15/16       0x8000000080adf180 

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

L6 Presenter

Thanks everyone for good advices. The setup we have is actualle an active/Passive HA pair and we did yesterday an upgrade from 7.0.8 to 7.0.13 to see it that helped. Unfortunately not:-) I checked the hardware pools and they wre ok.

When you enable threat are you then limited to 2 gbps throughput?

 

Here are statistics when load is 75% approx. Cant find anything peculiar.

 

 show running resource-monitor hour

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       0   0  55  83  65  89  66  89  67  90  69  90
       0   0  60  83  70  90  70  90  71  90  72  91
       0   0  61  85  72  92  73  92  74  92  75  92
       0   0  52  81  65  88  66  88  67  89  68  89
       0   0  20  70  30  71  31  69  33  67  36  68
       0   0   9  19  13  29  14  32  15  34  18  38
       0   0   8  54  10  86  11  86  13  87  15  87
       0   0   8  14  11  26  12  27  13  30  15  33
       0   0   9  15  11  31  12  32  13  33  15  35
       0   0   9  15  11  24  12  25  13  27  16  30
       0   0   9  19  11  28  12  29  13  31  15  33
       0   0   9  21  11  27  12  28  14  29  16  31
       0   0  10  20  12  24  13  24  15  25  17  28
       0   0  10  19  13  35  14  36  15  38  18  40
       0   0  11  44  14  39  15  40  17  40  19  42
       0   0  11  23  13  29  15  31  16  31  19  33
       0   0  12  23  15  29  17  30  18  36  21  39
       0   0  13  25  15  31  17  32  18  34  21  36
       0   0  12  30  15  40  17  38  18  38  21  40
       0   0  11  26  14  44  16  46  17  48  20  50
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0

Resource utilization (%) during last 24 hours:
session (average):
 15  16  16  15   7   4   3   3   3   3   3   4   4   4   4
  4   4   4   5   5   0   0   0   0
session (maximum):
 17  17  19  20  11   4   5   4   4   4   4   4   4   5   5
  5   5   5   5   7   0   0   0   0
packet buffer (average):
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet buffer (maximum):
  2   2   3   3   1   1   7   1   1   0   0   0   1   1   2
  1   2   2   1   2   0   0   0   0
packet descriptor (average):
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet descriptor (maximum):
  1   1   1   1   0   0   3   0   0   0   0   0   1   0   1
  0   1   1   0   1   0   0   0   0
packet descriptor (on-chip) (average):
  3   3   3   3   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   0   0   0   0
packet descriptor (on-chip) (maximum):
 38  37  43  44  23  16  12  10  12   5  11   4  16  11  11
 20  13  18  30  23   0   0   0   0

HA1(active)>

Hi TorC,

 

Consider using App override policy for sources/destinations that are mostly consuming your bandwidth.

 

You can use below KB it might help you:

 

https://live.paloaltonetworks.com/t5/Learning-Articles/Improving-Performance-of-HTTP-with-DSRI/ta-p/...

 

Regards,

Sharief

Regards,
Sharief
  • 4270 Views
  • 6 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!