09-20-2021 08:09 AM
Hi Team,
We came across a situation that our DP had just gone high at a time point till 100%. I checked with the system files and found some crashes have attached the screenshot for your reference.
Also all the software status where running which i checked later after the incident.
also find the output of show running resource monitor. This was taken during the time of issue. PAN OS Version is 9.0.13.
show running resource-monitor
[?1h=
Resource monitoring sampling data (per second):
CPU load sampling by group:
flow_lookup : 98%
flow_fastpath : 98%
flow_slowpath : 98%
flow_forwarding : 98%
flow_mgmt : 98%
flow_ctrl : 98%
nac_result : 98%
flow_np : 98%
dfa_result : 98%
module_internal : 98%
aho_result : 98%
zip_result : 98%
pktlog_forwarding : 98%
lwm : 0%
flow_host : 98%
CPU load (%) during last 60 seconds:
core 0 1 2 3 4 5
* 98 98 98 98 98
* 98 98 98 98 98
* 98 99 99 98 99
* 99 99 99 99 99
* 98 99 98 98 99
* 99 99 99 99 99
* 96 97 97 97 97
* 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 98 98 98 98 98
* 99 99 99 99 99
* 98 99 98 99 99
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 97 98 98 98 98
* 98 98 99 99 98
* 97 98 98 98 98
* 99 99 99 99 99
[7mlines 1-43[27m[K
[K * 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 98 98 98 98
* 99 99 99 99 99
* 99 99 99 99 99
* 97 98 98 98 98
* 95 96 96 96 96
* 95 97 97 97 97
* 94 96 96 96 96
* 99 99 99 99 99
* 98 99 99 99 99
* 97 97 98 97 97
* 96 97 97 97 97
* 96 97 97 97 98
* 98 99 99 99 99
* 99 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 98
* 98 99 99 99 99
* 96 97 97 97 97
* 97 98 98 98 98
* 98 98 98 98 98
* 96 97 97 97 97
* 98 98 98 98 98
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 98 99 99 99 99
* 99 99 99 99 99
* 98 98 98 98 98
Resource utilization (%) during last 60 seconds:
session:
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
[7mlines 44-86[27m[K
[K 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
packet buffer:
0 0 1 1 2 1 2 4 1 1 2 1 2 2 2
2 4 1 2 1 3 2 1 1 0 1 2 2 2 3
2 1 0 0 0 1 1 0 1 1 2 3 1 3 0
2 3 1 1 0 1 2 1 1 0 0 1 1 1 0
packet descriptor:
7 7 8 8 8 8 8 9 8 8 8 8 8 8 8
8 8 7 8 8 8 8 8 7 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
packet descriptor (on-chip):
6 9 16 20 58 41 43 62 22 16 54 32 49 60 61
34 57 13 50 38 71 63 28 12 9 37 44 65 50 63
42 15 10 10 8 32 24 7 24 33 56 62 36 52 9
44 58 12 31 9 23 58 15 25 2 7 23 36 20 9
Resource monitoring sampling data (per minute):
CPU load (%) during last 60 minutes:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 98 100 99 100 99 100 99 100 99 100
* * 97 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
[7mlines 87-129[27m[K
[K * * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 98 100 98 100 98 100 98 100 98 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
[7mlines 130-172[27m[K
[K * * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
Resource utilization (%) during last 60 minutes:
session (average):
15 15 15 15 15 16 16 16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16 16 16 16 16 16 17 17
17 17 17 17 18 18 18 18 18 18 17 17 17 17 16
16 17 16 16 16 16 17 17 17 17 17 16 17 15 15
session (maximum):
15 15 15 15 16 16 16 16 16 16 16 16 16 16 16
16 16 16 17 16 16 16 16 16 16 16 16 17 17 17
17 17 17 17 18 18 18 18 18 18 18 17 17 17 16
17 17 17 16 16 16 17 17 17 17 17 17 17 15 15
packet buffer (average):
3 2 2 3 3 3 3 3 4 4 5 3 3 3 4
4 4 4 5 3 5 5 4 5 5 4 5 5 6 4
5 7 5 6 8 6 7 7 7 6 8 7 6 8 7
6 5 6 6 6 6 6 6 6 7 6 7 6 2 1
packet buffer (maximum):
8 5 5 7 6 8 6 10 8 9 14 8 9 9 7
9 8 8 14 10 12 10 12 14 9 12 10 10 16 12
16 27 12 15 27 12 13 14 16 14 16 14 11 20 20
14 12 13 12 12 12 11 12 17 17 14 16 11 5 4
packet descriptor (average):
9 9 10 10 11 11 11 11 12 12 12 12 12 13 13
13 14 14 14 14 15 15 15 15 15 15 16 16 16 15
16 16 15 16 16 16 16 16 15 15 16 15 15 16 15
15 15 15 14 14 14 14 14 14 14 14 14 14 8 9
packet descriptor (maximum):
10 10 10 11 12 12 12 13 13 13 15 14 14 14 14
15 15 16 17 16 16 17 17 18 17 17 17 17 19 17
19 21 17 18 21 18 18 19 19 18 18 18 17 19 18
18 16 17 17 17 16 16 16 17 17 17 17 15 9 9
packet descriptor (on-chip) (average):
50 40 41 48 50 59 52 48 62 55 58 52 49 52 67
64 62 62 61 57 62 67 67 64 68 62 65 68 68 51
[7mlines 173-215[27m[K
[K 66 68 59 67 62 67 68 64 51 66 68 67 64 71 67
68 65 66 61 67 68 59 65 64 59 63 68 67 38 32
packet descriptor (on-chip) (maximum):
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
Resource monitoring sampling data (per hour):
CPU load (%) during last 24 hours:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 81 100 83 100 84 100 84 100 84 100
* * 37 62 46 70 46 70 46 70 46 70
* * 30 52 39 60 38 60 38 60 39 60
* * 30 51 38 58 38 58 38 59 38 59
* * 24 53 31 60 31 60 31 60 32 61
* * 12 53 16 58 15 58 15 58 16 58
* * 6 32 8 39 8 38 8 38 8 38
* * 8 38 10 44 10 44 10 44 10 43
* * 5 28 7 34 7 34 7 34 7 34
* * 3 25 4 31 4 31 4 30 4 30
* * 3 25 4 30 4 30 4 30 4 29
* * 3 25 4 28 4 28 4 28 4 28
* * 3 16 4 19 4 18 4 18 4 18
* * 4 23 5 29 5 29 5 29 5 29
* * 5 31 7 36 7 36 7 36 7 37
* * 6 31 7 35 7 34 7 35 7 35
* * 7 32 10 39 10 40 10 40 10 40
* * 7 29 10 33 10 34 10 33 10 33
* * 5 28 8 30 8 30 8 30 8 30
* * 6 21 8 28 8 27 8 28 8 28
* * 6 26 7 31 7 30 7 29 7 30
* * 6 20 8 24 7 25 7 24 8 24
* * 6 21 7 27 7 26 7 27 7 26
* * 7 32 9 37 9 37 9 37 9 37
Resource utilization (%) during last 24 hours:
session (average):
15 13 12 11 8 4 3 2 2 2 2 2 2 2 2
[7mlines 216-258[27m[K
[K 2 3 3 3 3 3 3 3 3
session (maximum):
26 14 13 11 11 6 3 3 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3
packet buffer (average):
4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet buffer (maximum):
89 1 2 1 1 1 0 1 0 0 1 0 1 1 0
0 1 1 1 1 0 0 1 0
packet descriptor (average):
7 1 0 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
34 1 1 1 1 1 0 1 0 0 1 0 0 0 0
0 0 0 1 1 0 0 0 1
packet descriptor (on-chip) (average):
36 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
73 37 70 11 53 19 3 9 3 13 9 8 44 5 4
4 8 4 10 56 3 6 41 3
Resource monitoring sampling data (per day):
CPU load (%) during last 7 days:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 5 39 7 45 7 44 7 45 7 43
* * 7 40 9 46 9 46 9 45 9 46
* * 16 60 20 66 20 66 20 66 20 66
* * 16 60 21 68 21 67 21 68 21 68
* * 17 94 21 96 21 96 21 96 22 96
* * 14 75 18 85 18 85 18 85 18 85
* * 14 100 18 100 18 100 17 100 18 100
Resource utilization (%) during last 7 days:
session (average):
2 3 6 7 6 6 6
session (maximum):
3 5 14 16 16 18 14
packet buffer (average):
0 0 0 0 0 0 0
[7mlines 259-301[27m[K
[Kpacket buffer (maximum):
2 3 3 2 3 3 3
packet descriptor (average):
0 0 0 0 0 0 0
packet descriptor (maximum):
1 1 1 1 3 3 3
packet descriptor (on-chip) (average):
2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
75 85 83 60 73 88 86
Resource monitoring sampling data (per week):
CPU load (%) during last 13 weeks:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 11 100 14 100 14 100 14 100 14 100
* * 11 75 15 80 14 80 14 80 15 80
* * 11 82 14 85 14 85 14 85 15 92
* * 7 69 9 90 9 90 9 90 9 93
* * 1 17 1 5 1 5 1 6 1 7
* * 1 16 1 6 1 6 1 6 1 7
* * 1 16 1 4 1 4 1 4 1 6
* * 1 19 1 10 1 10 1 11 1 13
* * 1 16 1 4 1 4 1 4 1 6
* * 1 17 1 4 1 5 1 5 1 6
* * 1 16 1 4 1 4 1 4 1 5
* * 1 17 1 4 1 5 1 5 1 7
* * 1 16 1 5 1 5 1 5 1 6
Resource utilization (%) during last 13 weeks:
session (average):
5 5 5 5 6 6 8 8 8 8 7 7 7
session (maximum):
18 14 14 19 15 25 26 23 22 22 20 21 20
packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet buffer (maximum):
3 4 8 5 0 0 0 0 0 0 0 0 0
packet descriptor (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
3 2 2 1 0 0 0 0 0 0 0 0 0
Please let me know if this is any sort of bug which could be fixed in any pan os versions
09-20-2021 12:58 PM
Did you look at the PAN-OS 9.0.14 release notes? There's a number of dataplane issues that were addressed in that release depending on your platform that you could be running into. You could have also seen a period of abnormal traffic load causing your dataplane load to reach platform limits.
Since this is more of a route cause investigation I would recommend engaging TAC so they actually have all of the relevant information to look at, rather than just guessing what the root cause was which is the only thing we can do with the information you provided.
09-21-2021 02:30 AM
I also came across with this details in syslog that there were multiple captive portal certificate and failed attempts i have attached the details for your reference.
17:58:50 info general general 0 Captive Portal Client certificate validation failed from 172.19.64.199.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:39 high general general 0 Dataplane under severe load
2021/09/20 14:38:21 info general general 0 Captive Portal Client certificate validation failed from 10.149.232.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:00 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20138.
2021/09/20 14:37:42 info general general 0 Captive Portal authentication succeeded for user: Mayur Ambekar <contractor> on 10.149.235.11, vsys1
2021/09/20 14:37:42 info general general 0 Captive Portal client certificate authentication successful from 10.149.235.11.
2021/09/20 14:36:21 info general general 0 Captive Portal authentication succeeded for user: vaibhav.desai@tatacommunications.com on 10.149.232.30, vsys1
2021/09/20 14:36:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.232.30.
2021/09/20 14:35:21 info general general 0 Captive Portal authentication succeeded for user: Navakanth.Pasupuleti@contractor.tatacommunications.com on 10.149.233.195, vsys1
2021/09/20 14:35:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.233.195.
2021/09/20 14:34:01 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20137.
2021/09/20 14:33:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:16 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:11 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:06 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:01 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:52 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:39 high general general 0 Dataplane under severe load
2021/09/20 14:31:20 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:19 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:17 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:08 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:04 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
Could this be causing the issue?
09-21-2021 03:25 AM
@Vijaygvasan This is very detailed issue description and as mentioned it is likely caused by a bug. Maybe it is better if you add all this to a support case.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
