topic Re: High dataplane CPU in General Topics

High dataplane CPU

Buck_Smooth — Fri, 03 Jun 2022 16:25:30 GMT

Last few days we have been experiencing high dp-cpu on all NPC simultaneously, specifically with flow_ctrl. The flow_ctrl process typically shows 3-10% CPU, but all of a sudden all NPC "DP slot x, dp 0 and 1" simultaneously jump to 30..50...80...100% and maintain that for 30-60 minutes, during which the firewall is basically down. Overall load is very low for PA-7050 (<10%). Clearing all sessions will restore all the core #2 CPU to normal, clearing only ICMP, TCP, UDP or ESP individually doesn't help, no other protocols are active.
"show running resource-monitor ingress-backlogs" and clearing those sessions doesn't restore CPU to normal either.
Full layer-3 directly connected interfaces to network (no L2/ARP/etc. to consider).
PanOS is 9.1.13h3, upgraded to 9.1.14 same issue.
Any suggestions?

Re: High dataplane CPU

reaper — Thu, 09 Jun 2022 08:56:20 GMT

if you run `show running resource-monitor` , what do the packet descriptors look like?

If theyre going up to 100 I may have seen a similar issue where a udp flood was consuming all packetbuffers. enabling packet buffer protection and tightening zone protection fixed the issue

Re: High dataplane CPU

OtakarKlier — Thu, 09 Jun 2022 14:57:27 GMT

Hello,

Could be a bug? However a support case could be in order.

Regards,

Re: High dataplane CPU

Buck_Smooth — Fri, 10 Jun 2022 15:07:18 GMT

This turned out to be an issue with a BGP peer that was suddenly sending us 15K routes which was putting us over the limit of 64K routes on the 7050. It was very difficult to figure out as there was no indication from any logs with the exception of a log we discovered regarding TCAM full, which pointed us in that direction. This could have easily been resolved had there been just a single message in the system logs regarding full route table.