High CPU Usage causing dropped packets..can't find what is causing it

drewdown — Sat, 16 May 2020 15:50:50 GMT

ACC isn't really telling me all that much except for a couple of end hosts with a fair amount of sessions so I blocked those but still doesn't seem to have helped whatsoever. I tried restarting the data plane as well to no avail. Can someone help me figure out what is going on here?

Resource monitoring sampling data (per second): CPU load sampling by group: flow_lookup : 100% flow_fastpath : 84% flow_slowpath : 100% flow_forwarding : 100% flow_mgmt : 20% flow_ctrl : 20% nac_result : 100% flow_np : 84% dfa_result : 100% module_internal : 100% aho_result : 100% zip_result : 100% pktlog_forwarding : 100% lwm : 0% flow_host : 80% CPU load (%) during last 60 seconds: core 0 1 2 3 4 5 * 20 100 100 100 100 * 19 100 100 100 100 * 20 100 100 100 100 * 22 100 100 100 100 * 22 100 100 100 100 * 22 100 100 100 100 * 19 100 100 100 100 * 18 81 80 80 81 * 19 91 91 91 92 * 18 99 99 99 99 * 17 100 100 100 100 * 17 100 100 100 100 * 19 100 100 100 100 * 22 100 100 100 100 * 22 100 100 100 100 * 22 100 100 100 100 * 24 100 100 100 100 * 22 98 98 98 98 * 17 88 88 88 88 * 15 88 87 87 88 * 14 84 84 84 84 * 14 81 81 82 82 * 12 65 65 66 66 * 14 77 77 77 77 * 18 92 92 92 92 * 19 87 87 87 87 * 18 88 88 88 88 * 18 98 98 98 98 * 17 100 100 100 100 * 17 100 100 100 100 * 18 100 100 100 100 * 19 100 100 100 100 * 19 100 100 100 100 * 21 100 100 100 100 * 22 100 100 100 100 * 23 100 100 100 100 * 21 100 100 100 100 * 19 99 99 99 99 * 12 58 58 58 58 * 19 88 88 88 88 * 26 100 100 100 100 * 24 100 100 100 100 * 22 100 100 100 100 * 21 100 100 100 100 * 19 100 100 100 100 * 17 100 100 100 100 * 15 100 100 100 100 * 15 100 100 100 100 * 15 100 100 100 100 * 16 98 98 98 98 * 16 94 94 94 94 * 15 84 84 84 84 * 13 71 71 71 71 * 15 75 75 75 75 * 12 68 69 69 69 * 17 81 81 81 81 * 23 97 97 97 97 * 25 100 100 100 100 * 26 100 100 100 100 * 24 100 100 100 100 Resource utilization (%) during last 60 seconds: session: 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 52 packet buffer: 81 91 91 91 85 70 47 18 36 79 93 93 93 63 46 36 26 10 5 3 3 2 2 2 2 4 3 73 93 93 93 91 91 90 90 74 59 35 8 58 86 90 90 89 90 89 90 93 93 67 20 3 2 2 2 2 30 70 82 87 packet descriptor: 20 23 23 23 21 18 12 5 1 18 23 24 24 16 12 9 7 3 2 1 1 1 1 1 1 1 1 19 24 24 24 23 23 23 23 19 15 9 2 15 22 23 23 23 23 23 23 23 24 17 5 1 1 1 1 1 8 18 21 22 packet descriptor (on-chip): 79 88 88 88 88 88 88 88 21 69 88 88 88 88 88 88 88 80 73 35 24 7 20 7 13 72 28 71 88 88 88 88 88 88 88 88 88 88 21 69 78 85 88 88 88 88 88 88 88 88 38 7 15 6 4 3 69 88 88 55 Resource monitoring sampling data (per minute): CPU load (%) during last 60 minutes: core 0 1 2 3 4 5 avg max avg max avg max avg max avg max avg max * * 19 30 95 100 95 100 95 100 95 100 * * 18 27 94 100 94 100 94 100 94 100 * * 19 25 97 100 97 100 97 100 97 100 * * 18 25 94 100 95 100 94 100 95 100 * * 18 25 93 100 93 100 93 100 93 100 * * 17 24 91 100 91 100 91 100 91 100 * * 19 31 94 100 94 100 94 100 94 100 * * 18 27 92 100 92 100 92 100 92 100 * * 18 24 94 100 94 100 94 100 94 100 * * 19 27 95 100 95 100 95 100 95 100 * * 18 26 95 100 95 100 95 100 95 100 * * 18 24 93 100 93 100 93 100 93 100 * * 19 29 94 100 94 100 94 100 94 100 * * 19 34 95 100 94 100 95 100 95 100 * * 18 31 93 100 93 100 93 100 93 100 * * 19 28 92 100 92 100 92 100 92 100 * * 19 28 94 100 94 100 94 100 94 100 * * 19 34 96 100 96 100 96 100 96 100 * * 18 25 90 100 90 100 90 100 91 100 * * 18 25 92 100 92 100 92 100 92 100 * * 17 24 85 100 85 100 85 100 85 100 * * 18 30 92 100 92 100 92 100 92 100 * * 19 35 89 100 89 100 89 100 89 100 * * 18 24 90 100 90 100 90 100 91 100 * * 18 23 92 100 92 100 92 100 92 100 * * 19 35 93 100 93 100 93 100 93 100 * * 21 41 87 100 87 100 87 100 87 100 * * 27 61 88 100 88 100 88 100 89 100 * * 32 54 87 100 87 100 88 100 88 100 * * 22 92 64 100 64 100 64 100 64 100 * * 2 9 1 9 1 9 1 9 1 9 * * 2 9 1 9 1 9 1 9 1 9 * * 2 9 1 11 1 11 1 11 1 11 * * 2 9 1 9 1 9 1 9 1 9 * * 2 9 1 9 1 10 1 9 1 9 * * 2 8 1 8 1 8 1 8 1 8 * * 2 9 1 9 1 10 1 9 1 10 * * 2 8 1 8 1 8 1 8 1 8 * * 2 8 1 8 1 8 1 8 1 8 * * 2 8 1 9 1 9 1 9 1 9 * * 2 10 1 11 1 11 1 11 1 11 * * 2 8 1 8 1 8 1 8 1 8 * * 2 11 1 11 1 12 1 11 1 12 * * 2 12 2 12 2 12 2 12 2 12 * * 2 9 2 10 2 10 2 10 2 10 * * 2 9 1 10 1 10 1 10 1 10 * * 2 9 2 10 2 10 2 10 2 10 * * 2 11 2 12 2 12 2 11 2 12 * * 2 11 2 12 2 12 2 12 2 12 * * 6 24 6 25 6 24 6 25 6 24 * * 2 9 1 9 1 9 1 9 1 9 * * 2 11 2 13 2 13 2 13 2 13 * * 2 9 2 10 2 10 2 10 2 10 * * 2 10 2 10 2 10 2 10 2 10 * * 2 10 2 10 2 10 2 10 2 10 * * 3 21 3 21 3 21 3 21 3 21 * * 2 10 2 10 2 10 2 10 2 10 * * 2 10 2 9 2 9 2 9 2 9 * * 2 9 2 9 2 9 2 9 2 9 * * 18 26 94 100 94 100 94 100 94 100 Resource utilization (%) during last 60 minutes: session (average): 52 53 53 53 53 53 53 53 53 54 55 56 56 58 58 59 60 61 69 70 71 71 70 64 65 69 72 72 73 60 54 54 54 55 56 59 59 60 62 63 63 63 64 66 68 70 70 70 70 72 65 67 70 71 71 66 47 48 48 52 session (maximum): 53 53 53 53 53 53 53 53 53 55 56 56 58 58 59 60 60 69 69 71 71 71 73 65 65 72 73 75 75 75 54 54 54 56 59 59 59 61 63 63 63 63 66 67 70 70 70 70 72 80 65 68 71 71 72 72 49 49 49 52 packet buffer (average): 52 56 62 57 59 47 55 50 57 52 48 60 52 68 47 58 60 62 54 59 51 60 52 58 55 54 51 48 45 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 55 packet buffer (maximum): 93 94 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 93 94 93 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 93 packet descriptor (average): 13 14 16 14 15 12 14 13 14 13 12 15 13 17 12 15 15 16 14 15 13 15 13 14 14 14 13 12 11 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 packet descriptor (maximum): 24 23 23 23 23 23 23 23 23 24 23 23 24 23 23 24 23 23 23 23 23 24 24 24 24 24 24 24 24 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24 packet descriptor (on-chip) (average): 65 67 69 68 65 56 65 59 58 63 64 61 66 72 64 68 70 70 64 69 58 72 65 67 63 68 68 61 61 48 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 67 packet descriptor (on-chip) (maximum): 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 88 2 2 2 4 2 2 2 2 2 2 2 2 2 2 2 2 5 2 2 19 2 2 2 2 2 2 2 2 2 88 Resource monitoring sampling data (per hour): CPU load (%) during last 24 hours: core 0 1 2 3 4 5 avg max avg max avg max avg max avg max avg max * * 2 92 3 100 3 100 3 100 3 100 * * 2 22 2 22 2 22 2 22 2 22 * * 2 12 2 12 2 12 2 12 2 12 * * 2 14 2 14 2 14 2 14 2 14 * * 2 13 2 13 2 13 2 13 2 13 * * 2 14 2 14 2 14 2 14 2 14 * * 2 18 2 18 2 18 2 18 2 18 * * 2 29 1 30 1 30 1 30 1 30 * * 2 31 1 31 1 31 1 31 1 31 * * 2 26 1 26 1 26 1 26 1 26 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 * * 1 1 0 1 0 1 0 1 0 1 Resource utilization (%) during last 24 hours: session (average): 55 47 48 48 48 47 47 27 39 36 4 4 4 4 4 4 4 4 4 4 4 4 4 4 session (maximum): 80 49 49 49 49 49 49 49 49 49 5 5 5 5 5 5 7 6 5 5 5 5 5 5 packet buffer (average): 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 packet buffer (maximum): 93 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 packet descriptor (average): 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 packet descriptor (maximum): 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 packet descriptor (on-chip) (average): 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 packet descriptor (on-chip) (maximum): 88 2 2 2 2 2 2 56 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 Resource monitoring sampling data (per day): CPU load (%) during last 7 days: core 0 1 2 3 4 5 avg max avg max avg max avg max avg max avg max * * 1 92 1 100 1 100 1 100 1 100 * * 1 3 0 3 0 3 0 3 0 3 * * 2 26 2 25 2 25 2 25 2 25 * * 1 34 1 35 1 34 1 35 1 34 * * 2 38 2 39 2 39 2 38 2 39 * * 1 5 0 5 0 5 0 5 0 5 * * 1 5 0 5 0 5 0 5 0 5 Resource utilization (%) during last 7 days: session (average): 21 4 38 12 20 4 4 session (maximum): 80 6 50 79 100 47 46 packet buffer (average): 0 0 0 0 0 0 0 packet buffer (maximum): 93 0 0 0 2 0 0 packet descriptor (average): 0 0 0 0 0 0 0 packet descriptor (maximum): 23 0 0 0 0 0 0 packet descriptor (on-chip) (average): 2 2 2 2 2 2 2 packet descriptor (on-chip) (maximum): 88 2 13 2 55 2 2 Resource monitoring sampling data (per week): CPU load (%) during last 13 weeks: core 0 1 2 3 4 5 avg max avg max avg max avg max avg max avg max * * 2 38 1 39 1 39 1 38 1 39 * * 3 21 2 11 2 13 2 13 2 14 * * 3 4 2 4 2 4 2 4 2 4 * * 3 12 2 12 2 12 2 12 2 12 * * 3 5 2 5 2 5 2 5 2 5 * * 3 4 2 5 2 5 2 5 2 5 * * 3 6 2 7 2 7 2 7 2 7 * * 3 26 2 26 2 26 2 26 2 26 * * 2 81 2 100 2 100 2 100 2 100 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Resource utilization (%) during last 13 weeks: session (average): 16 26 26 28 28 27 27 27 25 0 0 0 0 session (maximum): 100 49 31 49 40 36 47 47 100 0 0 0 0 packet buffer (average): 0 0 0 0 0 0 0 0 0 0 0 0 0 packet buffer (maximum): 2 0 0 0 0 0 0 4 94 0 0 0 0 packet descriptor (average): 0 0 0 0 0 0 0 0 0 0 0 0 0 packet descriptor (maximum): 0 0 0 0 0 0 0 1 25 0 0 0 0 packet descriptor (on-chip) (average): 2 2 2 2 2 2 2 2 2 0 0 0 0 packet descriptor (on-chip) (maximum): 55 2 2 2 2 2 2 56 88 0 0 0 0

Re: High CPU Usage causing dropped packets..can't find what is causing it

upelister — Sat, 16 May 2020 16:03:33 GMT

Hello,

Maybe Some sort of broadcast traffic hitting the FW, checking interface traffic via monitoring software can give clue.

Re: High CPU Usage causing dropped packets..can't find what is causing it

drewdown — Sat, 16 May 2020 16:21:39 GMT

I lied ACC told me exactly what was causing it I was just too dumb to enact the right policy to drop that traffic. Two offending hosts sending a ton of sessions outbound bringing the FW to a crawl. Once I got the right policy to deny all traffic from the offending hosts everything returned to normal.

Do PAs have something similar to ASA threat detection where it will shun/deny all traffic from a host if it sees x amount of traffic or sessions?

Re: High CPU Usage causing dropped packets..can't find what is causing it

upelister — Sat, 16 May 2020 16:42:54 GMT

Hi, Yes Palo Alto NGFW has;

Nice to hear everything gets normal.

Zone protection profiles. (should be applied on ingress zone, exhibit on last picture.)
DoS profiles. ( Should be used carefully classified or non-classified)
Packet buffer protection (Global setting should be applied on required zone)

Re: High CPU Usage causing dropped packets..can't find what is causing it

drewdown — Mon, 18 May 2020 19:05:59 GMT

I have zone protection but only on the untrust side but thinking now I need to do it on the trust as well.

topic High CPU Usage causing dropped packets..can't find what is causing it in General Topics

High CPU Usage causing dropped packets..can't find what is causing it

Re: High CPU Usage causing dropped packets..can't find what is causing it

Re: High CPU Usage causing dropped packets..can't find what is causing it

Re: High CPU Usage causing dropped packets..can't find what is causing it

Re: High CPU Usage causing dropped packets..can't find what is causing it