High dataplane CPU

[Buck_Smooth]

Last few days we have been experiencing high dp-cpu on all NPC simultaneously, specifically with flow_ctrl. The flow_ctrl process typically shows 3-10% CPU, but all of a sudden all NPC "DP slot x, dp 0 and 1" simultaneously jump to 30..50...80...100% and maintain that for 30-60 minutes, during which the firewall is basically down. Overall load is very low for PA-7050 (<10%). Clearing all sessions will restore all the core #2 CPU to normal, clearing only ICMP, TCP, UDP or ESP individually doesn't help, no other protocols are active.
"show running resource-monitor ingress-backlogs" and clearing those sessions doesn't restore CPU to normal either.
Full layer-3 directly connected interfaces to network (no L2/ARP/etc. to consider).
PanOS is 9.1.13h3, upgraded to 9.1.14 same issue.
Any suggestions?

[reaper]

if you run `show running resource-monitor` , what do the packet descriptors look like?

If theyre going up to 100 I may have seen a similar issue where a udp flood was consuming all packetbuffers. enabling packet buffer protection and tightening zone protection fixed the issue

[OtakarKlier]

Hello,

Could be a bug? However a support case could be in order.

Regards,

[Buck_Smooth · Accepted solution]

This turned out to be an issue with a BGP peer that was suddenly sending us 15K routes which was putting us over the limit of 64K routes on the 7050.  It was very difficult to figure out as there was no indication from any logs with the exception of a log we discovered regarding TCAM full, which pointed us in that direction.  This could have easily been resolved had there been just a single message in the system logs regarding full route table.