DP 100%

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

DP 100%

L3 Networker

Hi Team,

 

We came across a situation that our DP had just gone high at a time point till 100%. I checked with the system files and found some crashes have attached the screenshot for your reference.

 

Also all the software status where running which i checked later after the incident.

also find the output of show running resource monitor. This was taken during the time of issue. PAN OS Version is 9.0.13.

 

show running resource-monitor
[?1h=

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup : 98%
flow_fastpath : 98%
flow_slowpath : 98%
flow_forwarding : 98%
flow_mgmt : 98%
flow_ctrl : 98%
nac_result : 98%
flow_np : 98%
dfa_result : 98%
module_internal : 98%
aho_result : 98%
zip_result : 98%
pktlog_forwarding : 98%
lwm : 0%
flow_host : 98%

CPU load (%) during last 60 seconds:
core 0 1 2 3 4 5
* 98 98 98 98 98
* 98 98 98 98 98
* 98 99 99 98 99
* 99 99 99 99 99
* 98 99 98 98 99
* 99 99 99 99 99
* 96 97 97 97 97
* 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 98 98 98 98 98
* 99 99 99 99 99
* 98 99 98 99 99
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 97 98 98 98 98
* 98 98 99 99 98
* 97 98 98 98 98
* 99 99 99 99 99
lines 1-43
 * 98 99 99 99 99
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 99 99 99 99 99
* 100 100 100 100 100
* 99 99 99 99 99
* 98 98 98 98 98
* 99 99 99 99 99
* 99 99 99 99 99
* 97 98 98 98 98
* 95 96 96 96 96
* 95 97 97 97 97
* 94 96 96 96 96
* 99 99 99 99 99
* 98 99 99 99 99
* 97 97 98 97 97
* 96 97 97 97 97
* 96 97 97 97 98
* 98 99 99 99 99
* 99 100 100 100 100
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 99
* 99 99 99 99 99
* 98 99 99 99 98
* 98 99 99 99 99
* 96 97 97 97 97
* 97 98 98 98 98
* 98 98 98 98 98
* 96 97 97 97 97
* 98 98 98 98 98
* 98 99 99 99 99
* 98 98 98 98 98
* 97 98 98 98 98
* 98 99 99 99 99
* 99 99 99 99 99
* 98 98 98 98 98

Resource utilization (%) during last 60 seconds:
session:
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
lines 44-86
 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15

packet buffer:
0 0 1 1 2 1 2 4 1 1 2 1 2 2 2
2 4 1 2 1 3 2 1 1 0 1 2 2 2 3
2 1 0 0 0 1 1 0 1 1 2 3 1 3 0
2 3 1 1 0 1 2 1 1 0 0 1 1 1 0

packet descriptor:
7 7 8 8 8 8 8 9 8 8 8 8 8 8 8
8 8 7 8 8 8 8 8 7 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8 8 8

packet descriptor (on-chip):
6 9 16 20 58 41 43 62 22 16 54 32 49 60 61
34 57 13 50 38 71 63 28 12 9 37 44 65 50 63
42 15 10 10 8 32 24 7 24 33 56 62 36 52 9
44 58 12 31 9 23 58 15 25 2 7 23 36 20 9


Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 98 100 99 100 99 100 99 100 99 100
* * 97 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 99 100 99 100 99 100 99 100 99 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100
* * 98 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
lines 87-129
 * * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 98 100 98 100 98 100 98 100 98 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 100 100 100 100 100 100 100 100
* * 99 100 99 100 99 100 99 100 99 100
* * 99 100 99 100 99 100 99 100 99 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
* * 100 100 100 100 100 100 100 100 100 100
lines 130-172
 * * 98 100 98 100 98 100 98 100 98 100
* * 98 100 98 100 98 100 98 100 98 100

Resource utilization (%) during last 60 minutes:
session (average):
15 15 15 15 15 16 16 16 16 16 16 16 16 16 16
16 16 16 16 16 16 16 16 16 16 16 16 16 17 17
17 17 17 17 18 18 18 18 18 18 17 17 17 17 16
16 17 16 16 16 16 17 17 17 17 17 16 17 15 15

session (maximum):
15 15 15 15 16 16 16 16 16 16 16 16 16 16 16
16 16 16 17 16 16 16 16 16 16 16 16 17 17 17
17 17 17 17 18 18 18 18 18 18 18 17 17 17 16
17 17 17 16 16 16 17 17 17 17 17 17 17 15 15

packet buffer (average):
3 2 2 3 3 3 3 3 4 4 5 3 3 3 4
4 4 4 5 3 5 5 4 5 5 4 5 5 6 4
5 7 5 6 8 6 7 7 7 6 8 7 6 8 7
6 5 6 6 6 6 6 6 6 7 6 7 6 2 1

packet buffer (maximum):
8 5 5 7 6 8 6 10 8 9 14 8 9 9 7
9 8 8 14 10 12 10 12 14 9 12 10 10 16 12
16 27 12 15 27 12 13 14 16 14 16 14 11 20 20
14 12 13 12 12 12 11 12 17 17 14 16 11 5 4

packet descriptor (average):
9 9 10 10 11 11 11 11 12 12 12 12 12 13 13
13 14 14 14 14 15 15 15 15 15 15 16 16 16 15
16 16 15 16 16 16 16 16 15 15 16 15 15 16 15
15 15 15 14 14 14 14 14 14 14 14 14 14 8 9

packet descriptor (maximum):
10 10 10 11 12 12 12 13 13 13 15 14 14 14 14
15 15 16 17 16 16 17 17 18 17 17 17 17 19 17
19 21 17 18 21 18 18 19 19 18 18 18 17 19 18
18 16 17 17 17 16 16 16 17 17 17 17 15 9 9

packet descriptor (on-chip) (average):
50 40 41 48 50 59 52 48 62 55 58 52 49 52 67
64 62 62 61 57 62 67 67 64 68 62 65 68 68 51
lines 173-215
 66 68 59 67 62 67 68 64 51 66 68 67 64 71 67
68 65 66 61 67 68 59 65 64 59 63 68 67 38 32

packet descriptor (on-chip) (maximum):
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71
71 71 71 71 71 71 71 71 71 71 71 71 71 71 71


Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 81 100 83 100 84 100 84 100 84 100
* * 37 62 46 70 46 70 46 70 46 70
* * 30 52 39 60 38 60 38 60 39 60
* * 30 51 38 58 38 58 38 59 38 59
* * 24 53 31 60 31 60 31 60 32 61
* * 12 53 16 58 15 58 15 58 16 58
* * 6 32 8 39 8 38 8 38 8 38
* * 8 38 10 44 10 44 10 44 10 43
* * 5 28 7 34 7 34 7 34 7 34
* * 3 25 4 31 4 31 4 30 4 30
* * 3 25 4 30 4 30 4 30 4 29
* * 3 25 4 28 4 28 4 28 4 28
* * 3 16 4 19 4 18 4 18 4 18
* * 4 23 5 29 5 29 5 29 5 29
* * 5 31 7 36 7 36 7 36 7 37
* * 6 31 7 35 7 34 7 35 7 35
* * 7 32 10 39 10 40 10 40 10 40
* * 7 29 10 33 10 34 10 33 10 33
* * 5 28 8 30 8 30 8 30 8 30
* * 6 21 8 28 8 27 8 28 8 28
* * 6 26 7 31 7 30 7 29 7 30
* * 6 20 8 24 7 25 7 24 8 24
* * 6 21 7 27 7 26 7 27 7 26
* * 7 32 9 37 9 37 9 37 9 37

Resource utilization (%) during last 24 hours:
session (average):
15 13 12 11 8 4 3 2 2 2 2 2 2 2 2
lines 216-258
 2 3 3 3 3 3 3 3 3
session (maximum):
26 14 13 11 11 6 3 3 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3
packet buffer (average):
4 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet buffer (maximum):
89 1 2 1 1 1 0 1 0 0 1 0 1 1 0
0 1 1 1 1 0 0 1 0
packet descriptor (average):
7 1 0 1 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
34 1 1 1 1 1 0 1 0 0 1 0 0 0 0
0 0 0 1 1 0 0 0 1
packet descriptor (on-chip) (average):
36 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
73 37 70 11 53 19 3 9 3 13 9 8 44 5 4
4 8 4 10 56 3 6 41 3

Resource monitoring sampling data (per day):

CPU load (%) during last 7 days:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 5 39 7 45 7 44 7 45 7 43
* * 7 40 9 46 9 46 9 45 9 46
* * 16 60 20 66 20 66 20 66 20 66
* * 16 60 21 68 21 67 21 68 21 68
* * 17 94 21 96 21 96 21 96 22 96
* * 14 75 18 85 18 85 18 85 18 85
* * 14 100 18 100 18 100 17 100 18 100

Resource utilization (%) during last 7 days:
session (average):
2 3 6 7 6 6 6
session (maximum):
3 5 14 16 16 18 14
packet buffer (average):
0 0 0 0 0 0 0
lines 259-301
packet buffer (maximum):
2 3 3 2 3 3 3
packet descriptor (average):
0 0 0 0 0 0 0
packet descriptor (maximum):
1 1 1 1 3 3 3
packet descriptor (on-chip) (average):
2 2 2 2 2 2 2
packet descriptor (on-chip) (maximum):
75 85 83 60 73 88 86

Resource monitoring sampling data (per week):

CPU load (%) during last 13 weeks:
core 0 1 2 3 4 5
avg max avg max avg max avg max avg max avg max
* * 11 100 14 100 14 100 14 100 14 100
* * 11 75 15 80 14 80 14 80 15 80
* * 11 82 14 85 14 85 14 85 15 92
* * 7 69 9 90 9 90 9 90 9 93
* * 1 17 1 5 1 5 1 6 1 7
* * 1 16 1 6 1 6 1 6 1 7
* * 1 16 1 4 1 4 1 4 1 6
* * 1 19 1 10 1 10 1 11 1 13
* * 1 16 1 4 1 4 1 4 1 6
* * 1 17 1 4 1 5 1 5 1 6
* * 1 16 1 4 1 4 1 4 1 5
* * 1 17 1 4 1 5 1 5 1 7
* * 1 16 1 5 1 5 1 5 1 6

Resource utilization (%) during last 13 weeks:
session (average):
5 5 5 5 6 6 8 8 8 8 7 7 7
session (maximum):
18 14 14 19 15 25 26 23 22 22 20 21 20
packet buffer (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet buffer (maximum):
3 4 8 5 0 0 0 0 0 0 0 0 0
packet descriptor (average):
0 0 0 0 0 0 0 0 0 0 0 0 0
packet descriptor (maximum):
3 2 2 1 0 0 0 0 0 0 0 0 0

 

Please let me know if this is any sort of bug which could be fixed in any pan os versions

3 REPLIES 3

Cyber Elite
Cyber Elite

@Vijaygvasan,

Did you look at the PAN-OS 9.0.14 release notes? There's a number of dataplane issues that were addressed in that release  depending on your platform that you could be running into. You could have also seen a period of abnormal traffic load causing your dataplane load to reach platform limits. 

Since this is more of a route cause investigation I would recommend engaging TAC so they actually have all of the relevant information to look at, rather than just guessing what the root cause was which is the only thing we can do with the information you provided. 

I also came across with this details in syslog that there were multiple captive portal certificate and failed attempts i have attached the details for your reference.

 

17:58:50 info general general 0 Captive Portal Client certificate validation failed from 172.19.64.199.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:39 high general general 0 Dataplane under severe load
2021/09/20 14:38:21 info general general 0 Captive Portal Client certificate validation failed from 10.149.232.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:38:00 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20138.
2021/09/20 14:37:42 info general general 0 Captive Portal authentication succeeded for user: Mayur Ambekar <contractor> on 10.149.235.11, vsys1
2021/09/20 14:37:42 info general general 0 Captive Portal client certificate authentication successful from 10.149.235.11.
2021/09/20 14:36:21 info general general 0 Captive Portal authentication succeeded for user: vaibhav.desai@tatacommunications.com on 10.149.232.30, vsys1
2021/09/20 14:36:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.232.30.
2021/09/20 14:35:21 info general general 0 Captive Portal authentication succeeded for user: Navakanth.Pasupuleti@contractor.tatacommunications.com on 10.149.233.195, vsys1
2021/09/20 14:35:21 info general general 0 Captive Portal client certificate authentication successful from 10.149.233.195.
2021/09/20 14:34:01 info url-fil upgrade 0 PAN-DB was upgraded to version 20210920.20137.
2021/09/20 14:33:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:39 high general general 0 Dataplane under severe load
2021/09/20 14:32:16 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:11 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:06 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:32:01 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:52 info general general 0 Captive Portal Client certificate validation failed from 10.149.234.35.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:39 high general general 0 Dataplane under severe load
2021/09/20 14:31:20 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:19 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:17 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:12 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:08 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain
2021/09/20 14:31:04 info general general 0 Captive Portal Client certificate validation failed from 10.149.235.204.Certificate does not belong to the Cert Profile chain

 

 

Could this be causing the issue?

L4 Transporter

@Vijaygvasan This is very detailed issue description and as mentioned it is likely caused by a bug. Maybe it is better if you add all this to a support case. 

  • 2171 Views
  • 3 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!