01-17-2017 06:22 AM - edited 01-17-2017 06:23 AM
We have a PA-3050 on softwarre 7.0.8 and are struggling with CPU dataplase issues during normal business hours. 80 % load is shown by dashboard in webgui. Session count is approx. 80 000 when issue occurs.
show system resources follow - only says that logrcvr uses 5% and nothing special is elsewhere noticed.
Running command show system statistics session during CPU high load reports approx 500-700 Mbps througput average.
Total supported througput with threat should be 2Gbps and without threat 4 Gbps. Im unsure what happends when threat is enabled, is supported througput then limited to 2 Gbps?
Any advice?
Regards
\\Tor
01-17-2017 06:45 AM
It sounds like that is running a little high with the number of sessions that you have open. That being said it would be interesting to know what your CPS is, if that is reaching near max then these numbers make sense. As far as the logrcvr goes it really depends on how your loggging is setup on your security policies, without looking at more numbers I can't really tell you if that is higher than it should be or not.
01-17-2017 01:17 PM - edited 01-17-2017 01:32 PM
Hi,
For this sort of issue, l think better to raise a TAC case. They do have a special technics to determine what is actually causing high CPU usage. Or it could be just simple overload or even normal behaviour when handling that much sessions. Did you observe this behaviour all the time on any previous PAN-OS? Could be even a bug in the software:
7.0.10 has a fix but not sure if you facing this bug or not:
Thx,
Myky
01-18-2017 03:02 AM
show system resources is to display Management plane resource usage
show running resource-monitor will give you the dataplane core utilisation:
admin@myNGFW> show running resource-monitor second
Resource monitoring sampling data (per second):
CPU load sampling by group:
flow_lookup : 0%
flow_fastpath : 0%
flow_slowpath : 0%
flow_forwarding : 0%
flow_mgmt : 1%
flow_ctrl : 1%
nac_result : 0%
flow_np : 0%
dfa_result : 0%
module_internal : 0%
aho_result : 0%
zip_result : 0%
pktlog_forwarding : 0%
lwm : 0%
flow_host : 1%
CPU load (%) during last 60 seconds:
core 0 1 2 3 4 5
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
* 1 0 0 0 0
additionally, show session info will give you an overview of packetrates, bandwidth usage etc:
admin@myNGFW> show session info target-dp: *.dp0 -------------------------------------------------------------------------------- Number of sessions supported: 262142 Number of active sessions: 2 Number of active TCP sessions: 0 Number of active UDP sessions: 2 Number of active ICMP sessions: 0 Number of active GTPc sessions: 0 Number of active GTPu sessions: 0 Number of active BCAST sessions: 0 Number of active MCAST sessions: 0 Number of active predict sessions: 0 Session table utilization: 0% Number of sessions created since bootup: 25852 Packet rate: 32/s Throughput: 15 kbps New connection establish rate: 0 cps
When threat is enabled, the average speed we measured in lab was about 2gbps but there is no limitation. some scenarios may have a little less, others will have a little more. it depends on your traffic patern
you could check your hard-software pools to see if any of them are getting depleted
if any of them have a very low number to the left of the / (or even 0), that pool is depleted, which could be a hint at what is causing your issues
admin@myNGFW> debug dataplane pool statistics Hardware Pools [ 0] Packet Buffers : 11337/11468 0x80000000e7a8f000 [ 1] Work Queue Entries : 228233/229376 0x80000000e5e8f000 [ 2] Output Buffers : 4547/4556 0x80000000eea8d000 [ 3] DFA Result : 3999/4000 0x800000010fa18000 [ 4] Timer Buffers : 4093/4096 0x800000010f618000 [ 5] LWM Pool : 1024/1024 0x800000010f5d8000 [ 6] ZIP Commands : 1023/1024 0x800000010fe00000 [ 7] Dma Cmd Buffers : 1016/1024 0x80000000e5d8f000 Software Pools [ 0] Shared Pool 24 ( 24): 557429/560000 0x80000000cc122f80 [ 1] Shared Pool 32 ( 32): 528594/530000 0x80000000cd017280 [ 2] Shared Pool 40 ( 40): 70000/70000 0x80000000ce249700 [ 3] Shared Pool 192 ( 192): 1791975/1795000 0x80000000ce539780 [ 4] Shared Pool 256 ( 256): 139993/140000 0x80000000e34bf180 [ 5] software packet buffer 0 ( 512): 32223/32768 0x8000000095356680 [ 6] software packet buffer 1 ( 1024): 32768/32768 0x8000000096376780 [ 7] software packet buffer 2 ( 2048): 81376/81920 0x8000000098396880 [ 8] software packet buffer 3 (33280): 20480/20480 0x80000000a23e6980 [ 9] software packet buffer 4 (66048): 304/304 0x80000000cadfaa80 [10] ZIP Results ( 184): 1024/1024 0x8000000083f9e300 [11] CTD AV Block ( 1024): 32/32 0x8000000082bdc380 [12] Regex Results (11544): 8000/8000 0x8000000056202880 22 [13] SSH Handshake State ( 6512): 64/64 0x80000000808e8680 [14] SSH State ( 3200): 512/512 0x800000008094e480 [15] TCP host connections ( 176): 15/16 0x8000000080adf180
01-18-2017 03:08 AM
Add to the comment above this KB might help:
https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/FeaturedArticles/article-id/147
01-19-2017 04:19 AM - edited 01-19-2017 04:20 AM
Thanks everyone for good advices. The setup we have is actualle an active/Passive HA pair and we did yesterday an upgrade from 7.0.8 to 7.0.13 to see it that helped. Unfortunately not:-) I checked the hardware pools and they wre ok.
When you enable threat are you then limited to 2 gbps throughput?
Here are statistics when load is 75% approx. Cant find anything peculiar.
show running resource-monitor hour
Resource monitoring sampling data (per hour):
CPU load (%) during last 24 hours:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
0 0 55 83 65 89 66 89 67 90 69 90
0 0 60 83 70 90 70 90 71 90 72 91
0 0 61 85 72 92 73 92 74 92 75 92
0 0 52 81 65 88 66 88 67 89 68 89
0 0 20 70 30 71 31 69 33 67 36 68
0 0 9 19 13 29 14 32 15 34 18 38
0 0 8 54 10 86 11 86 13 87 15 87
0 0 8 14 11 26 12 27 13 30 15 33
0 0 9 15 11 31 12 32 13 33 15 35
0 0 9 15 11 24 12 25 13 27 16 30
0 0 9 19 11 28 12 29 13 31 15 33
0 0 9 21 11 27 12 28 14 29 16 31
0 0 10 20 12 24 13 24 15 25 17 28
0 0 10 19 13 35 14 36 15 38 18 40
0 0 11 44 14 39 15 40 17 40 19 42
0 0 11 23 13 29 15 31 16 31 19 33
0 0 12 23 15 29 17 30 18 36 21 39
0 0 13 25 15 31 17 32 18 34 21 36
0 0 12 30 15 40 17 38 18 38 21 40
0 0 11 26 14 44 16 46 17 48 20 50
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
Resource utilization (%) during last 24 hours:
session (average):
15 16 16 15 7 4 3 3 3 3 3 4 4 4 4
4 4 4 5 5 0 0 0 0
session (maximum):
17 17 19 20 11 4 5 4 4 4 4 4 4 5 5
5 5 5 5 7 0 0 0 0
packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet buffer (maximum):
2 2 3 3 1 1 7 1 1 0 0 0 1 1 2
1 2 2 1 2 0 0 0 0
packet descriptor (average):
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
1 1 1 1 0 0 3 0 0 0 0 0 1 0 1
0 1 1 0 1 0 0 0 0
packet descriptor (on-chip) (average):
3 3 3 3 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 0 0 0 0
packet descriptor (on-chip) (maximum):
38 37 43 44 23 16 12 10 12 5 11 4 16 11 11
20 13 18 30 23 0 0 0 0
HA1(active)>
02-04-2017 10:36 PM
Hi TorC,
Consider using App override policy for sources/destinations that are mostly consuming your bandwidth.
You can use below KB it might help you:
Regards,
Sharief
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
