topic Re: DP 100% in General Topics

DP 100%

Vijaygvasan — Mon, 20 Sep 2021 15:09:49 GMT

Hi Team,

We came across a situation that our DP had just gone high at a time point till 100%. I checked with the system files and found some crashes have attached the screenshot for your reference.

Also all the software status where running which i checked later after the incident.

also find the output of show running resource monitor. This was taken during the time of issue. PAN OS Version is 9.0.13.

show running resource-monitor
[?1h=

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup : 98%
flow_fastpath : 98%
flow_slowpath : 98%
flow_forwarding : 98%
flow_mgmt : 98%
flow_ctrl : 98%
nac_result : 98%
flow_np : 98%
dfa_result : 98%
module_internal : 98%
aho_result : 98%
zip_result : 98%
pktlog_forwarding : 98%
lwm : 0%
flow_host : 98%

CPU load (%) during last 60 seconds:
core 0 1 2 3 4 5
* 98 98 98 98 98
* 98 98 98 98 98
* 98 99 99 98 99
* 99 99 99 99 99
* 98 99 98 98 99
* 99 99 99 99 99
* 96 97 97 97 97
* 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 98 98 98 98 98
* 99 99 99 99 99
* 98 99 98 99 99
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 97 98 98 98 98
* 98 98 99 99 98
* 97 98 98 98 98
* 99 99 99 99 99
[7mlines 1-43[27m[K
[K * 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 98 98 98 98
* 99 99 99 99 99
* 99 99 99 99 99
* 97 98 98 98 98
* 95 96 96 96 96
* 95 97 97 97 97
* 94 96 96 96 96
* 99 99 99 99 99
* 98 99 99 99 99
* 97 97 98 97 97
* 96 97 97 97 97
* 96 97 97 97 98
* 98 99 99 99 99
* 99 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 98
* 98 99 99 99 99
* 96 97 97 97 97
* 97 98 98 98 98
* 98 98 98 98 98
* 96 97 97 97 97
* 98 98 98 98 98
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 98 99 99 99 99
* 99 99 99 99 99
* 98 98 98 98 98

Resource utilization (%) during last 60 seconds:
session:
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
[7mlines 44-86[27m[K
[K 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15

packet buffer:
0 0 1 1 2 1 2 4 1 1 2 1 2 2 2
2 4 1 2 1 3 2 1 1 0 1 2 2 2 3
2 1 0 0 0 1 1 0 1 1 2 3 1 3 0
2 3 1 1 0 1 2 1 1 0 0 1 1 1 0

packet descriptor:
7 7 8 8 8 8 8 9 8 8 8 8 8 8 8
8 8 7 8 8 8 8 8 7 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8

packet descriptor (on-chip):
6 9 16 20 58 41 43 62 22 16 54 32 49 60 61
34 57 13 50 38 71 63 28 12 9 37 44 65 50 63
42 15 10 10 8 32 24 7 24 33 56 62 36 52 9
44 58 12 31 9 23 58 15 25 2 7 23 36 20 9

Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 98 100 99 100 99 100 99 100 99 100
* * 97 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
[7mlines 87-129[27m[K
[K * * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 98 100 98 100 98 100 98 100 98 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
[7mlines 130-172[27m[K
[K * * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100

Resource utilization (%) during last 60 minutes:
session (average):
15 15 15 15 15 16 16 16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16 16 16 16 16 16 17 17
17 17 17 17 18 18 18 18 18 18 17 17 17 17 16
16 17 16 16 16 16 17 17 17 17 17 16 17 15 15

session (maximum):
15 15 15 15 16 16 16 16 16 16 16 16 16 16 16
16 16 16 17 16 16 16 16 16 16 16 16 17 17 17
17 17 17 17 18 18 18 18 18 18 18 17 17 17 16
17 17 17 16 16 16 17 17 17 17 17 17 17 15 15

packet buffer (average):
3 2 2 3 3 3 3 3 4 4 5 3 3 3 4
4 4 4 5 3 5 5 4 5 5 4 5 5 6 4
5 7 5 6 8 6 7 7 7 6 8 7 6 8 7
6 5 6 6 6 6 6 6 6 7 6 7 6 2 1

packet buffer (maximum):
8 5 5 7 6 8 6 10 8 9 14 8 9 9 7
9 8 8 14 10 12 10 12 14 9 12 10 10 16 12
16 27 12 15 27 12 13 14 16 14 16 14 11 20 20
14 12 13 12 12 12 11 12 17 17 14 16 11 5 4

packet descriptor (average):
9 9 10 10 11 11 11 11 12 12 12 12 12 13 13
13 14 14 14 14 15 15 15 15 15 15 16 16 16 15
16 16 15 16 16 16 16 16 15 15 16 15 15 16 15
15 15 15 14 14 14 14 14 14 14 14 14 14 8 9

packet descriptor (maximum):
10 10 10 11 12 12 12 13 13 13 15 14 14 14 14
15 15 16 17 16 16 17 17 18 17 17 17 17 19 17
19 21 17 18 21 18 18 19 19 18 18 18 17 19 18
18 16 17 17 17 16 16 16 17 17 17 17 15 9 9

packet descriptor (on-chip) (average):
50 40 41 48 50 59 52 48 62 55 58 52 49 52 67
64 62 62 61 57 62 67 67 64 68 62 65 68 68 51
[7mlines 173-215[27m[K
[K 66 68 59 67 62 67 68 64 51 66 68 67 64 71 67
68 65 66 61 67 68 59 65 64 59 63 68 67 38 32

packet descriptor (on-chip) (maximum):
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71

Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 81 100 83 100 84 100 84 100 84 100
* * 37 62 46 70 46 70 46 70 46 70
* * 30 52 39 60 38 60 38 60 39 60
* * 30 51 38 58 38 58 38 59 38 59
* * 24 53 31 60 31 60 31 60 32 61
* * 12 53 16 58 15 58 15 58 16 58
* * 6 32 8 39 8 38 8 38 8 38
* * 8 38 10 44 10 44 10 44 10 43
* * 5 28 7 34 7 34 7 34 7 34
* * 3 25 4 31 4 31 4 30 4 30
* * 3 25 4 30 4 30 4 30 4 29
* * 3 25 4 28 4 28 4 28 4 28
* * 3 16 4 19 4 18 4 18 4 18
* * 4 23 5 29 5 29 5 29 5 29
* * 5 31 7 36 7 36 7 36 7 37
* * 6 31 7 35 7 34 7 35 7 35
* * 7 32 10 39 10 40 10 40 10 40
* * 7 29 10 33 10 34 10 33 10 33
* * 5 28 8 30 8 30 8 30 8 30
* * 6 21 8 28 8 27 8 28 8 28
* * 6 26 7 31 7 30 7 29 7 30
* * 6 20 8 24 7 25 7 24 8 24
* * 6 21 7 27 7 26 7 27 7 26
* * 7 32 9 37 9 37 9 37 9 37

Resource utilization (%) during last 24 hours:
session (average):
15 13 12 11 8 4 3 2 2 2 2 2 2 2 2
[7mlines 216-258[27m[K
[K 2 3 3 3 3 3 3 3 3
session (maximum):
26 14 13 11 11 6 3 3 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3
packet buffer (average):
4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet buffer (maximum):
89 1 2 1 1 1 0 1 0 0 1 0 1 1 0
0 1 1 1 1 0 0 1 0
packet descriptor (average):
7 1 0 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
34 1 1 1 1 1 0 1 0 0 1 0 0 0 0
0 0 0 1 1 0 0 0 1
packet descriptor (on-chip) (average):
36 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
73 37 70 11 53 19 3 9 3 13 9 8 44 5 4
4 8 4 10 56 3 6 41 3

Resource monitoring sampling data (per day):

CPU load (%) during last 7 days:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 5 39 7 45 7 44 7 45 7 43
* * 7 40 9 46 9 46 9 45 9 46
* * 16 60 20 66 20 66 20 66 20 66
* * 16 60 21 68 21 67 21 68 21 68
* * 17 94 21 96 21 96 21 96 22 96
* * 14 75 18 85 18 85 18 85 18 85
* * 14 100 18 100 18 100 17 100 18 100

Resource utilization (%) during last 7 days:
session (average):
2 3 6 7 6 6 6
session (maximum):
3 5 14 16 16 18 14
packet buffer (average):
0 0 0 0 0 0 0
[7mlines 259-301[27m[K
[Kpacket buffer (maximum):
2 3 3 2 3 3 3
packet descriptor (average):
0 0 0 0 0 0 0
packet descriptor (maximum):
1 1 1 1 3 3 3
packet descriptor (on-chip) (average):
2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
75 85 83 60 73 88 86

Resource monitoring sampling data (per week):

CPU load (%) during last 13 weeks:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 11 100 14 100 14 100 14 100 14 100
* * 11 75 15 80 14 80 14 80 15 80
* * 11 82 14 85 14 85 14 85 15 92
* * 7 69 9 90 9 90 9 90 9 93
* * 1 17 1 5 1 5 1 6 1 7
* * 1 16 1 6 1 6 1 6 1 7
* * 1 16 1 4 1 4 1 4 1 6
* * 1 19 1 10 1 10 1 11 1 13
* * 1 16 1 4 1 4 1 4 1 6
* * 1 17 1 4 1 5 1 5 1 6
* * 1 16 1 4 1 4 1 4 1 5
* * 1 17 1 4 1 5 1 5 1 7
* * 1 16 1 5 1 5 1 5 1 6

Resource utilization (%) during last 13 weeks:
session (average):
5 5 5 5 6 6 8 8 8 8 7 7 7
session (maximum):
18 14 14 19 15 25 26 23 22 22 20 21 20
packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet buffer (maximum):
3 4 8 5 0 0 0 0 0 0 0 0 0
packet descriptor (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
3 2 2 1 0 0 0 0 0 0 0 0 0

Please let me know if this is any sort of bug which could be fixed in any pan os versions

Re: DP 100%

BPry — Mon, 20 Sep 2021 19:58:22 GMT

@Vijaygvasan,

Did you look at the PAN-OS 9.0.14 release notes? There's a number of dataplane issues that were addressed in that release depending on your platform that you could be running into. You could have also seen a period of abnormal traffic load causing your dataplane load to reach platform limits.

Since this is more of a route cause investigation I would recommend engaging TAC so they actually have all of the relevant information to look at, rather than just guessing what the root cause was which is the only thing we can do with the information you provided.

Re: DP 100%

Vijaygvasan — Tue, 21 Sep 2021 09:30:38 GMT

I also came across with this details in syslog that there were multiple captive portal certificate and failed attempts i have attached the details for your reference.

17:58:50 info general general 0 Captive Portal Client certificate validation failed from 172.19.64.199.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:39 high general general 0 Dataplane under severe load
2021/09/20 14:38:21 info general general 0 Captive Portal Client certificate validation failed from 10.149.232.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:00 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20138.
2021/09/20 14:37:42 info general general 0 Captive Portal authentication succeeded for user: Mayur Ambekar <contractor> on 10.149.235.11, vsys1
2021/09/20 14:37:42 info general general 0 Captive Portal client certificate authentication successful from 10.149.235.11.
2021/09/20 14:36:21 info general general 0 Captive Portal authentication succeeded for user: vaibhav.desai@tatacommunications.com on 10.149.232.30, vsys1
2021/09/20 14:36:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.232.30.
2021/09/20 14:35:21 info general general 0 Captive Portal authentication succeeded for user: Navakanth.Pasupuleti@contractor.tatacommunications.com on 10.149.233.195, vsys1
2021/09/20 14:35:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.233.195.
2021/09/20 14:34:01 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20137.
2021/09/20 14:33:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:16 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:11 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:06 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:01 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:52 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:39 high general general 0 Dataplane under severe load
2021/09/20 14:31:20 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:19 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:17 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:08 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:04 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain

Could this be causing the issue?

Re: DP 100%

batd2 — Tue, 21 Sep 2021 10:25:35 GMT

@Vijaygvasan This is very detailed issue description and as mentioned it is likely caused by a bug. Maybe it is better if you add all this to a support case.