PA-3060 Dataplane

[sfranchina]

Hi Fellow Palo-Alto-ers,

Hoping this amazing community can help me shed some light on something!

I have a PA-3060 running PAN-OS 8.0.2. I am wondering what would be considered high dataplane CPU utilisation for this particular platform i.e. at what percentage should I start to see traffic drops! I read 80%+ is concerning according to Palo KB articles.

This unit cruises along at anywhere between 30-70% dataplane CPU (all cores) when the unit is basically sleeping (~16000 sessions) and throughput according to "show session info" is a low 10M tops, and interface utilisation (according to "show system state browser") is not much higher. It's currently a low traffic period.

I've been told this unit grinds to a halt during high traffic periods (of which I am yet to witness to gather outputs, so I don't know what high traffic really means as yet). If I am interpreting "show running resource monitor" correctly, the unit frequently hits the "max" figures for all cores on dataplane even in low traffic situations.

Just wondering is this normal for this particular platform (i.e. to max out CPU with such low throughput and low session count?). Could traffic bursts cause the DP CPU to max out? I intend to run these commands again when the high traffic period resumes.

Any pointers would be greatly appreciated!

Kind Regards - Shane

*****

show running resource-monitor

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup : 37%
flow_fastpath : 34%
flow_slowpath : 37%
flow_forwarding : 37%
flow_mgmt : 22%
flow_ctrl : 22%
nac_result : 37%
flow_np : 34%
dfa_result : 37%
module_internal : 37%
aho_result : 37%
zip_result : 37%
pktlog_forwarding : 32%
lwm : 0%
flow_host : 35%

CPU load (%) during last 60 seconds:
core 0 1 2 3 4 5
* 31 38 34 38 39
* 21 36 40 44 31
* 24 38 38 37 34
* 21 28 35 27 29
* 24 39 39 39 35
* 47 48 45 48 44
* 36 39 41 34 36
* 25 42 41 31 36
* 43 60 56 53 57
* 45 62 54 59 57
* 39 51 46 53 46
* 59 71 72 72 70
* 70 83 78 80 79
* 49 61 60 59 60
* 59 65 63 65 64
* 59 62 62 65 63
* 40 45 44 52 45
* 27 39 35 43 40
* 26 39 38 35 40
* 33 48 49 45 53
* 29 43 43 35 42
* 29 39 45 36 43
* 25 42 42 39 38
* 28 47 48 45 46
* 27 40 45 38 37
* 30 37 44 40 38
* 34 47 48 45 44
* 34 37 41 41 40
* 38 43 49 45 44
* 32 43 46 40 38
* 29 39 40 37 29
* 37 43 48 46 40
* 60 65 67 65 67
* 56 64 62 63 65
* 55 65 61 63 64
* 55 67 65 66 65
* 45 54 59 61 55
* 38 47 53 57 50
* 36 41 46 47 42
* 44 47 55 52 46
* 54 62 68 65 61
* 48 57 60 56 56
* 38 47 46 45 48
* 50 65 62 64 66
* 55 65 64 64 65
* 43 55 54 52 52
* 36 54 53 49 49
* 53 66 64 59 60
* 68 79 76 77 75
* 52 71 66 69 68
* 46 66 62 63 64
* 52 67 67 66 67
* 50 65 63 63 64
* 56 70 69 67 68
* 54 67 66 64 64
* 42 51 50 45 46
* 39 46 46 41 45
* 42 51 49 48 50
* 33 41 42 38 41
* 27 33 35 37 34

Resource utilization (%) during last 60 seconds:
session:
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

packet buffer:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

packet descriptor:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

packet descriptor (on-chip):
2 2 2 3 2 15 7 5 4 4 2 2 6 4 10
9 8 5 3 2 2 2 8 5 6 4 2 2 2 3
7 5 2 17 11 3 3 2 2 2 4 3 2 6 5
3 3 2 9 20 12 3 13 11 8 9 3 5 5 2

Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 46 76 56 86 56 86 55 85 55 87
* * 41 77 52 87 51 83 50 86 50 83
* * 39 73 51 84 51 81 51 81 51 81
* * 37 63 49 76 49 78 48 75 48 81
* * 46 67 54 79 54 79 54 80 54 82
* * 35 72 44 85 44 84 43 85 43 86
* * 37 62 48 75 48 73 47 74 48 72
* * 43 69 55 79 54 83 53 81 53 79
* * 33 67 43 72 43 74 41 72 42 75
* * 35 60 47 70 46 70 45 67 46 70
* * 36 68 48 71 47 71 47 69 47 72
* * 34 59 48 77 47 75 46 75 46 77
* * 33 55 45 67 43 67 43 66 43 63
* * 34 61 47 70 47 72 47 72 46 74
* * 45 73 59 87 59 86 59 86 58 84
* * 44 73 57 85 56 83 57 82 57 83
* * 42 67 54 84 53 81 53 81 53 80
* * 35 65 46 70 46 73 45 73 46 73
* * 34 51 43 60 43 66 42 58 42 60
* * 31 58 40 70 40 66 40 67 40 64
* * 37 71 47 77 47 75 47 76 46 73
* * 36 64 46 77 46 76 46 72 46 70
* * 36 66 45 75 45 76 44 72 44 72
* * 28 51 39 62 39 62 39 59 38 57
* * 36 65 48 78 47 81 47 79 48 77
* * 34 62 45 67 44 64 44 67 44 68
* * 37 63 50 82 49 84 49 82 49 81
* * 41 71 54 75 53 77 51 73 52 77
* * 34 54 48 69 46 67 46 66 46 67
* * 29 45 40 58 39 59 39 57 40 58
* * 35 61 50 77 48 74 47 71 47 73
* * 40 66 52 77 52 74 51 77 51 77
* * 46 81 58 90 57 88 56 91 56 90
* * 40 73 52 82 51 82 50 84 50 81
* * 35 56 49 64 48 62 46 61 46 67
* * 39 64 52 73 52 76 50 74 50 72
* * 38 93 49 91 46 89 46 89 46 88
* * 30 50 43 63 42 60 41 61 41 60
* * 37 60 49 69 48 69 47 68 47 72
* * 42 69 52 78 52 76 51 78 51 77
* * 42 71 52 83 51 79 51 80 51 80
* * 36 56 47 65 47 64 46 63 45 65
* * 36 58 46 67 46 72 46 68 45 67
* * 32 68 41 68 41 70 40 71 41 69
* * 33 71 44 77 44 79 43 77 43 77
* * 36 65 46 72 46 72 45 71 45 70
* * 40 81 52 89 51 90 51 90 51 90
* * 35 73 47 77 47 81 45 76 46 75
* * 35 65 46 77 45 72 44 74 43 69
* * 48 98 60 98 59 98 58 99 58 98
* * 45 79 59 82 58 80 57 78 57 79
* * 49 75 63 89 62 89 61 89 61 87
* * 47 86 58 93 57 92 57 91 56 94
* * 36 77 49 91 48 90 47 91 48 92
* * 51 88 61 94 61 92 59 91 60 91
* * 43 68 55 79 56 77 55 78 54 79
* * 41 75 54 82 53 82 52 80 53 81
* * 41 65 53 79 53 78 52 81 52 77
* * 46 77 55 81 56 83 55 84 55 84
* * 37 56 46 70 46 67 45 66 45 68

Resource utilization (%) during last 60 minutes:
session (average):
4 4 4 3 3 3 4 4 4 4 3 3 4 4 4
4 4 4 3 4 4 3 3 3 3 3 3 3 3 3
3 3 3 3 3 4 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 4 3 3 4 4

session (maximum):
4 4 4 4 4 3 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 3 3 4 4 3 3 3 3
3 4 3 3 3 4 3 3 3 3 3 3 3 3 3
3 3 3 3 3 4 3 4 3 3 4 3 4 4 4

packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

packet buffer (maximum):
1 1 1 0 1 1 1 1 1 1 2 1 1 1 1
1 1 1 1 1 1 0 1 0 1 1 1 1 1 0
0 0 1 1 0 1 1 1 1 1 1 1 1 1 0
1 1 1 1 1 1 1 1 1 1 1 1 1 1 0

packet descriptor (average):
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 2 2 1 1 1 1 1 1 1 1 1
1 1 2 2 3 3 3 3 4 3 3 2 2 1 1

packet descriptor (maximum):
2 1 1 1 1 1 2 2 1 1 2 1 1 1 2
2 1 1 1 1 1 1 2 2 1 2 2 2 1 1
1 2 2 2 3 2 2 2 1 1 1 1 1 1 1
2 1 2 3 3 4 4 4 4 4 3 3 2 1 2

packet descriptor (on-chip) (average):
6 6 6 4 6 4 4 5 4 4 6 4 5 6 6
6 5 4 4 3 5 4 5 4 5 4 5 6 5 3
4 4 5 5 4 5 4 3 5 5 4 6 3 4 3
6 6 5 5 7 6 6 7 4 9 6 5 5 5 4

packet descriptor (on-chip) (maximum):
45 40 42 14 41 26 16 19 19 34 80 19 23 22 48
19 18 22 14 17 31 13 37 13 17 25 23 26 31 14
11 16 31 42 11 29 19 22 32 40 15 24 33 21 12
34 27 55 26 34 24 17 40 17 41 29 25 19 20 11

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 38 98 49 98 49 98 48 99 48 98
* * 33 100 44 96 43 94 42 95 42 94
* * 26 96 38 91 37 91 36 89 36 89
* * 13 99 18 65 18 66 18 99 18 90
* * 4 66 6 72 6 66 6 69 6 72
* * 2 39 3 34 3 42 3 65 3 67
* * 2 49 3 54 3 85 3 54 3 50
* * 2 54 3 62 3 46 3 79 3 44
* * 2 39 3 35 3 38 3 36 3 51
* * 2 33 3 33 3 28 3 34 3 31
* * 2 40 3 37 3 38 3 63 3 36
* * 2 44 3 46 3 46 3 46 3 45
* * 2 62 3 61 3 65 3 57 3 59
* * 2 62 4 98 4 98 4 98 4 98
* * 2 40 4 51 4 44 4 40 4 39
* * 3 84 4 46 4 52 4 43 4 47
* * 3 53 7 88 7 59 6 49 6 46
* * 4 79 8 100 7 100 7 100 7 100
* * 7 43 11 55 10 80 10 56 10 51
* * 14 72 22 79 22 72 21 74 21 96
* * 23 93 34 98 33 98 33 98 33 98
* * 32 87 45 91 44 91 44 92 44 91
* * 36 96 48 97 48 100 47 99 47 100
* * 30 91 41 94 40 94 40 94 40 95

Resource utilization (%) during last 24 hours:
session (average):
3 3 3 2 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 2 3 4 4 4
session (maximum):
4 4 4 3 2 1 1 1 1 2 1 1 1 1 1
2 2 2 2 3 4 5 5 5
packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet buffer (maximum):
1 1 1 2 1 1 0 0 0 0 0 0 0 1 0
1 1 1 1 1 1 6 1 1
packet descriptor (average):
2 1 1 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1
packet descriptor (maximum):
4 2 2 1 1 1 1 0 0 1 0 0 0 1 0
1 1 1 1 1 2 3 2 2
packet descriptor (on-chip) (average):
5 4 4 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 3 3 5 5 4
packet descriptor (on-chip) (maximum):
55 60 57 71 18 19 8 7 11 11 7 3 10 31 11
29 30 34 36 45 56 84 61 47

Resource monitoring sampling data (per day):

CPU load (%) during last 7 days:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 12 98 17 100 17 100 17 100 17 100
* * 13 97 18 100 18 100 18 100 18 100
* * 13 100 18 100 18 100 17 100 17 100
* * 2 89 3 93 3 89 3 92 3 95
* * 3 96 4 80 4 78 4 69 4 77
* * 14 100 20 100 19 100 19 100 19 100
* * * * * * * * * * * *

Resource utilization (%) during last 7 days:
session (average):
2 2 2 1 1 2 0
session (maximum):
5 5 5 1 2 7 0
packet buffer (average):
0 0 0 0 0 0 0
packet buffer (maximum):
6 2 2 1 1 3 0
packet descriptor (average):
1 0 0 0 0 0 0
packet descriptor (maximum):
3 4 2 0 1 6 0
packet descriptor (on-chip) (average):
3 3 3 2 2 3 0
packet descriptor (on-chip) (maximum):
84 71 71 29 39 72 0

=====
show session info

target-dp: *.dp0
--------------------------------------------------------------------------------
Number of sessions supported: 524286
Number of allocated sessions: 16360
Number of active TCP sessions: 11584
Number of active UDP sessions: 4385
Number of active ICMP sessions: 27
Number of active GTPc sessions: 0
Number of active GTPu sessions: 0
Number of pending GTPu sessions: 0
Number of active BCAST sessions: 0
Number of active MCAST sessions: 0
Number of active predict sessions: 11
Session table utilization: 3%
Number of sessions created since bootup: 52191444
Packet rate: 256/s
Throughput: 195 kbps
New connection establish rate: 170 cps
--------------------------------------------------------------------------------
Session timeout
TCP default timeout: 3600 secs
TCP session timeout before SYN-ACK received: 5 secs
TCP session timeout before 3-way handshaking: 10 secs
TCP half-closed session timeout: 120 secs
TCP session timeout in TIME_WAIT: 15 secs
TCP session timeout for unverified RST: 30 secs
UDP default timeout: 30 secs
ICMP default timeout: 6 secs
other IP default timeout: 30 secs
Captive Portal session timeout: 30 secs
Session timeout in discard state:
TCP: 90 secs, UDP: 60 secs, other IP protocols: 60 secs
--------------------------------------------------------------------------------
Session accelerated aging: True
Accelerated aging threshold: 80% of utilization
Scaling factor: 2 X
--------------------------------------------------------------------------------
Session setup
TCP - reject non-SYN first packet: True
Hardware session offloading: True
IPv6 firewalling: True

[reaper]

you can gain a little more insight by polling the global counters to see if something specific is 'high' and causing a rise in CPU (like processing many zipped files through fileblocking/av scanning, or doing a lot of ssl decryption etc.

> show counter global filter delta yes (do this at least twice so the delta kicks in)

a high cpu load should not immediately be a cause for concern. Most firewalls will be able to run at 100% without breaking a sweat or dropping a single packet.

What you need to watch for is packet descriptors. If those surpass 80%, you've got a bottleneck and may indeed see packetloss and latency

The processes listed at the top will routinely be running at 100% once a decent amount of traffic is flowing, as those are 'pre spun up'

[sfranchina]

Cheers reaper - good to know! I have seen some high "max" packet descriptor values > 80%, but pretty low "average" values. Perhaps transient traffic spikes.

I'll keep an eye on those global counters. Also it's entirely possible that we are decrypting a LOT of sessions - in quiet times I saw 4k sessions, but during peak load... we could very well be doing near the platform max. I'll keep an eye on that too - thanks for your help!