dp-monitor.log entry meanings?

[Sec101]

We are running 8.1.4 on a pair of 820's, and having been having some issues with certain traffic. After some help and digging, we are seeing random hits an (entry below) for the Max % utilization for 100% in one part of the file, but NOT on show running resource monitor OR SNMP. The Avg hovers around 4-20%. When I watch the GUI, or Pan(w)achrome - I never see the data plane spike over 5-20%. It never hits 100%. However, I do see the (entry below) in the "tail follow yes mp-log dp-monitor.log". SNMP shows no spikes on data plane, polling the OID of 1.3.6.1.2.1.25.3.3.1.2.2 shows no spikes in the DP cpu and a "show running resource monitor" shows no spikes . Out put below

2019-01-24 16:50:02.340 -0600 --- cpu
Last 180 seconds
Avg (%) Max (%)
1 3
Load Avg:
4.01 4.20 4.39 5/704 27550
-----------------------------

2019-01-24 16:53:02.360 -0600 --- cpu
Last 180 seconds
Avg (%) Max (%)
20 100
Load Avg:
4.42 4.23 4.37 5/705 28123

(show running resource monitor below)

CPU load (%) during last 5 minutes:
core 0 1 2 3 4 5 6 7
avg max avg max avg max avg max avg max avg max avg max avg max
* * 0 0 1 1 0 0 0 0 * * * * * *
* * 0 0 1 1 0 0 0 0 * * * * * *
* * 0 0 1 1 0 0 0 0 * * * * * *
* * 0 1 1 1 0 0 0 0 * * * * * *
* * 0 0 1 1 0 0 0 0 * * * * * *

Resource utilization (%) during last 5 minutes:
session (average):
0 0 0 0 0
session (maximum):
0 0 0 0 0
packet buffer (average):
0 0 0 0 0
packet buffer (maximum):
0 0 0 0 0
packet descriptor (average):
0 0 0 0 0
packet descriptor (maximum):
0 0 0 0 0
packet descriptor (on-chip) (average):
5 5 5 5 5
packet descriptor (on-chip) (maximum):
5 5 5 5 5

What am I missing here, and what are the first two entries for the Last 180 seconds, as I'm unable to correlate that with the "show running resource monitor minute/second last" cmd over that same time period for utilization.

[Raido_Rattameister]

Start by turning on high DP log.

Device > Setup > Logging and Reporting Settings > Log Export and Reporting
Check "Enable Log on High DP Load"

This will log event into System log when DP is under load.

Then you can try to find pattern or correlate with traffic going through firewall at that time.

Unknown traffic is heavy on dataplane so if you have backup traffic that encrypts packets that fw can't inspect it might make sense to create application override for etc.

[Sec101]

Yup, did that already, never saw an entry in the GUI for high data plane.  however, I can go back in the dp-monitor.log and verify that there were mutlipe times where the MAX% under the "Last 180 seconds" was at 100%, but the AVG is way lower.   

Is this indeed saying that the DP processor was at 100%?

[Raido_Rattameister]

Maybe those links will help a bit.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClVpCAK

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRTCA0