High CPU Usage causing dropped packets..can't find what is causing it

Reply
Highlighted
L4 Transporter

High CPU Usage causing dropped packets..can't find what is causing it

ACC isn't really telling me all that much except for a couple of end hosts with a fair amount of sessions so I blocked those but still doesn't seem to have helped whatsoever.  I tried restarting the data plane as well to no avail.  Can someone help me figure out what is going on here?  

 

Resource monitoring sampling data (per second):

CPU load sampling by group:
flow_lookup                    :   100%
flow_fastpath                  :    84%
flow_slowpath                  :   100%
flow_forwarding                :   100%
flow_mgmt                      :    20%
flow_ctrl                      :    20%
nac_result                     :   100%
flow_np                        :    84%
dfa_result                     :   100%
module_internal                :   100%
aho_result                     :   100%
zip_result                     :   100%
pktlog_forwarding              :   100%
lwm                            :     0%
flow_host                      :    80%

CPU load (%) during last 60 seconds:
core   0   1   2   3   4   5
       *  20 100 100 100 100
       *  19 100 100 100 100
       *  20 100 100 100 100
       *  22 100 100 100 100
       *  22 100 100 100 100
       *  22 100 100 100 100
       *  19 100 100 100 100
       *  18  81  80  80  81
       *  19  91  91  91  92
       *  18  99  99  99  99
       *  17 100 100 100 100
       *  17 100 100 100 100
       *  19 100 100 100 100
       *  22 100 100 100 100
       *  22 100 100 100 100
       *  22 100 100 100 100
       *  24 100 100 100 100
       *  22  98  98  98  98
       *  17  88  88  88  88
       *  15  88  87  87  88
       *  14  84  84  84  84
       *  14  81  81  82  82
       *  12  65  65  66  66
       *  14  77  77  77  77
       *  18  92  92  92  92
       *  19  87  87  87  87
       *  18  88  88  88  88
       *  18  98  98  98  98
       *  17 100 100 100 100
       *  17 100 100 100 100
       *  18 100 100 100 100
       *  19 100 100 100 100
       *  19 100 100 100 100
       *  21 100 100 100 100
       *  22 100 100 100 100
       *  23 100 100 100 100
       *  21 100 100 100 100
       *  19  99  99  99  99
       *  12  58  58  58  58
       *  19  88  88  88  88
       *  26 100 100 100 100
       *  24 100 100 100 100
       *  22 100 100 100 100
       *  21 100 100 100 100
       *  19 100 100 100 100
       *  17 100 100 100 100
       *  15 100 100 100 100
       *  15 100 100 100 100
       *  15 100 100 100 100
       *  16  98  98  98  98
       *  16  94  94  94  94
       *  15  84  84  84  84
       *  13  71  71  71  71
       *  15  75  75  75  75
       *  12  68  69  69  69
       *  17  81  81  81  81
       *  23  97  97  97  97
       *  25 100 100 100 100
       *  26 100 100 100 100
       *  24 100 100 100 100

Resource utilization (%) during last 60 seconds:
session:
 52  52  52  52  52  52  52  52  52  52  52  52  52  52  52
 52  52  52  52  52  52  52  52  52  52  52  52  52  52  52
 52  52  52  52  52  52  52  52  52  52  52  52  52  52  52
 52  52  52  52  52  52  52  52  52  52  52  52  52  52  52

packet buffer:
 81  91  91  91  85  70  47  18  36  79  93  93  93  63  46
 36  26  10   5   3   3   2   2   2   2   4   3  73  93  93
 93  91  91  90  90  74  59  35   8  58  86  90  90  89  90
 89  90  93  93  67  20   3   2   2   2   2  30  70  82  87

packet descriptor:
 20  23  23  23  21  18  12   5   1  18  23  24  24  16  12
  9   7   3   2   1   1   1   1   1   1   1   1  19  24  24
 24  23  23  23  23  19  15   9   2  15  22  23  23  23  23
 23  23  23  24  17   5   1   1   1   1   1   8  18  21  22

packet descriptor (on-chip):
 79  88  88  88  88  88  88  88  21  69  88  88  88  88  88
 88  88  80  73  35  24   7  20   7  13  72  28  71  88  88
 88  88  88  88  88  88  88  88  21  69  78  85  88  88  88
 88  88  88  88  88  38   7  15   6   4   3  69  88  88  55


Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       *   *  19  30  95 100  95 100  95 100  95 100
       *   *  18  27  94 100  94 100  94 100  94 100
       *   *  19  25  97 100  97 100  97 100  97 100
       *   *  18  25  94 100  95 100  94 100  95 100
       *   *  18  25  93 100  93 100  93 100  93 100
       *   *  17  24  91 100  91 100  91 100  91 100
       *   *  19  31  94 100  94 100  94 100  94 100
       *   *  18  27  92 100  92 100  92 100  92 100
       *   *  18  24  94 100  94 100  94 100  94 100
       *   *  19  27  95 100  95 100  95 100  95 100
       *   *  18  26  95 100  95 100  95 100  95 100
       *   *  18  24  93 100  93 100  93 100  93 100
       *   *  19  29  94 100  94 100  94 100  94 100
       *   *  19  34  95 100  94 100  95 100  95 100
       *   *  18  31  93 100  93 100  93 100  93 100
       *   *  19  28  92 100  92 100  92 100  92 100
       *   *  19  28  94 100  94 100  94 100  94 100
       *   *  19  34  96 100  96 100  96 100  96 100
       *   *  18  25  90 100  90 100  90 100  91 100
       *   *  18  25  92 100  92 100  92 100  92 100
       *   *  17  24  85 100  85 100  85 100  85 100
       *   *  18  30  92 100  92 100  92 100  92 100
       *   *  19  35  89 100  89 100  89 100  89 100
       *   *  18  24  90 100  90 100  90 100  91 100
       *   *  18  23  92 100  92 100  92 100  92 100
       *   *  19  35  93 100  93 100  93 100  93 100
       *   *  21  41  87 100  87 100  87 100  87 100
       *   *  27  61  88 100  88 100  88 100  89 100
       *   *  32  54  87 100  87 100  88 100  88 100
       *   *  22  92  64 100  64 100  64 100  64 100
       *   *   2   9   1   9   1   9   1   9   1   9
       *   *   2   9   1   9   1   9   1   9   1   9
       *   *   2   9   1  11   1  11   1  11   1  11
       *   *   2   9   1   9   1   9   1   9   1   9
       *   *   2   9   1   9   1  10   1   9   1   9
       *   *   2   8   1   8   1   8   1   8   1   8
       *   *   2   9   1   9   1  10   1   9   1  10
       *   *   2   8   1   8   1   8   1   8   1   8
       *   *   2   8   1   8   1   8   1   8   1   8
       *   *   2   8   1   9   1   9   1   9   1   9
       *   *   2  10   1  11   1  11   1  11   1  11
       *   *   2   8   1   8   1   8   1   8   1   8
       *   *   2  11   1  11   1  12   1  11   1  12
       *   *   2  12   2  12   2  12   2  12   2  12
       *   *   2   9   2  10   2  10   2  10   2  10
       *   *   2   9   1  10   1  10   1  10   1  10
       *   *   2   9   2  10   2  10   2  10   2  10
       *   *   2  11   2  12   2  12   2  11   2  12
       *   *   2  11   2  12   2  12   2  12   2  12
       *   *   6  24   6  25   6  24   6  25   6  24
       *   *   2   9   1   9   1   9   1   9   1   9
       *   *   2  11   2  13   2  13   2  13   2  13
       *   *   2   9   2  10   2  10   2  10   2  10
       *   *   2  10   2  10   2  10   2  10   2  10
       *   *   2  10   2  10   2  10   2  10   2  10
       *   *   3  21   3  21   3  21   3  21   3  21
       *   *   2  10   2  10   2  10   2  10   2  10
       *   *   2  10   2   9   2   9   2   9   2   9
       *   *   2   9   2   9   2   9   2   9   2   9
       *   *  18  26  94 100  94 100  94 100  94 100

Resource utilization (%) during last 60 minutes:
session (average):
 52  53  53  53  53  53  53  53  53  54  55  56  56  58  58
 59  60  61  69  70  71  71  70  64  65  69  72  72  73  60
 54  54  54  55  56  59  59  60  62  63  63  63  64  66  68
 70  70  70  70  72  65  67  70  71  71  66  47  48  48  52

session (maximum):
 53  53  53  53  53  53  53  53  53  55  56  56  58  58  59
 60  60  69  69  71  71  71  73  65  65  72  73  75  75  75
 54  54  54  56  59  59  59  61  63  63  63  63  66  67  70
 70  70  70  72  80  65  68  71  71  72  72  49  49  49  52

packet buffer (average):
 52  56  62  57  59  47  55  50  57  52  48  60  52  68  47
 58  60  62  54  59  51  60  52  58  55  54  51  48  45  30
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0   0   0   0   0   0  55

packet buffer (maximum):
 93  94  93  93  93  93  93  93  93  93  93  93  93  93  93
 93  93  93  93  93  93  93  93  93  93  93  93  93  94  93
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   1   0   0   0   0   0   0   0   0   0  93

packet descriptor (average):
 13  14  16  14  15  12  14  13  14  13  12  15  13  17  12
 15  15  16  14  15  13  15  13  14  14  14  13  12  11   8
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0   0   0   0   0   0  14

packet descriptor (maximum):
 24  23  23  23  23  23  23  23  23  24  23  23  24  23  23
 24  23  23  23  23  23  24  24  24  24  24  24  24  24  23
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0   0   0   0   0   0  24

packet descriptor (on-chip) (average):
 65  67  69  68  65  56  65  59  58  63  64  61  66  72  64
 68  70  70  64  69  58  72  65  67  63  68  68  61  61  48
  2   2   2   2   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2   2   2   2   2   2  67

packet descriptor (on-chip) (maximum):
 88  88  88  88  88  88  88  88  88  88  88  88  88  88  88
 88  88  88  88  88  88  88  88  88  88  88  88  88  88  88
  2   2   2   4   2   2   2   2   2   2   2   2   2   2   2
  2   5   2   2  19   2   2   2   2   2   2   2   2   2  88


Resource monitoring sampling data (per hour):

CPU load (%) during last 24 hours:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       *   *   2  92   3 100   3 100   3 100   3 100
       *   *   2  22   2  22   2  22   2  22   2  22
       *   *   2  12   2  12   2  12   2  12   2  12
       *   *   2  14   2  14   2  14   2  14   2  14
       *   *   2  13   2  13   2  13   2  13   2  13
       *   *   2  14   2  14   2  14   2  14   2  14
       *   *   2  18   2  18   2  18   2  18   2  18
       *   *   2  29   1  30   1  30   1  30   1  30
       *   *   2  31   1  31   1  31   1  31   1  31
       *   *   2  26   1  26   1  26   1  26   1  26
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1
       *   *   1   1   0   1   0   1   0   1   0   1

Resource utilization (%) during last 24 hours:
session (average):
 55  47  48  48  48  47  47  27  39  36   4   4   4   4   4
  4   4   4   4   4   4   4   4   4
session (maximum):
 80  49  49  49  49  49  49  49  49  49   5   5   5   5   5
  5   7   6   5   5   5   5   5   5
packet buffer (average):
  1   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet buffer (maximum):
 93   0   0   0   0   0   0   2   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet descriptor (average):
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet descriptor (maximum):
 23   0   0   0   0   0   0   0   0   0   0   0   0   0   0
  0   0   0   0   0   0   0   0   0
packet descriptor (on-chip) (average):
  3   2   2   2   2   2   2   2   2   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2
packet descriptor (on-chip) (maximum):
 88   2   2   2   2   2   2  56   3   2   2   2   2   2   2
  2   2   2   2   2   2   2   2   2

Resource monitoring sampling data (per day):

CPU load (%) during last 7 days:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       *   *   1  92   1 100   1 100   1 100   1 100
       *   *   1   3   0   3   0   3   0   3   0   3
       *   *   2  26   2  25   2  25   2  25   2  25
       *   *   1  34   1  35   1  34   1  35   1  34
       *   *   2  38   2  39   2  39   2  38   2  39
       *   *   1   5   0   5   0   5   0   5   0   5
       *   *   1   5   0   5   0   5   0   5   0   5

Resource utilization (%) during last 7 days:
session (average):
 21   4  38  12  20   4   4
session (maximum):
 80   6  50  79 100  47  46
packet buffer (average):
  0   0   0   0   0   0   0
packet buffer (maximum):
 93   0   0   0   2   0   0
packet descriptor (average):
  0   0   0   0   0   0   0
packet descriptor (maximum):
 23   0   0   0   0   0   0
packet descriptor (on-chip) (average):
  2   2   2   2   2   2   2
packet descriptor (on-chip) (maximum):
 88   2  13   2  55   2   2

Resource monitoring sampling data (per week):

CPU load (%) during last 13 weeks:
core    0       1       2       3       4       5
     avg max avg max avg max avg max avg max avg max
       *   *   2  38   1  39   1  39   1  38   1  39
       *   *   3  21   2  11   2  13   2  13   2  14
       *   *   3   4   2   4   2   4   2   4   2   4
       *   *   3  12   2  12   2  12   2  12   2  12
       *   *   3   5   2   5   2   5   2   5   2   5
       *   *   3   4   2   5   2   5   2   5   2   5
       *   *   3   6   2   7   2   7   2   7   2   7
       *   *   3  26   2  26   2  26   2  26   2  26
       *   *   2  81   2 100   2 100   2 100   2 100
       *   *   *   *   *   *   *   *   *   *   *   *
       *   *   *   *   *   *   *   *   *   *   *   *
       *   *   *   *   *   *   *   *   *   *   *   *
       *   *   *   *   *   *   *   *   *   *   *   *

Resource utilization (%) during last 13 weeks:
session (average):
 16  26  26  28  28  27  27  27  25   0   0   0   0
session (maximum):
100  49  31  49  40  36  47  47 100   0   0   0   0
packet buffer (average):
  0   0   0   0   0   0   0   0   0   0   0   0   0
packet buffer (maximum):
  2   0   0   0   0   0   0   4  94   0   0   0   0
packet descriptor (average):
  0   0   0   0   0   0   0   0   0   0   0   0   0
packet descriptor (maximum):
  0   0   0   0   0   0   0   1  25   0   0   0   0
packet descriptor (on-chip) (average):
  2   2   2   2   2   2   2   2   2   0   0   0   0
packet descriptor (on-chip) (maximum):
 55   2   2   2   2   2   2  56  88   0   0   0   0

   

Highlighted
L2 Linker

Hello,

 

Maybe Some sort of broadcast traffic hitting the FW, checking interface traffic via monitoring software can give clue.

UP
Highlighted
L4 Transporter

I lied ACC told me exactly what was causing it I was just too dumb to enact the right policy to drop that traffic.  Two offending hosts sending a ton of sessions outbound bringing the FW to a crawl.   Once I got the right policy to deny all traffic from the offending hosts everything returned to normal. 

 

Do PAs have something similar to ASA threat detection where it will shun/deny all traffic from a host if it sees x amount of traffic or sessions?  

Highlighted
L2 Linker

Hi, Yes Palo Alto NGFW has;

Nice to hear everything gets normal.

  • Zone protection profiles. (should be applied on ingress zone, exhibit on last picture.)
  • DoS profiles. ( Should be used carefully classified or non-classified)
  • Packet buffer protection (Global setting should be applied on required zone)zp_1.JPG

     

    zp_2.JPG

     

    dp1.jpg

     

    packect_buffer.JPG

     

    zone_settings.JPG

     

UP
Highlighted
L4 Transporter

I have zone protection but only on the untrust side but thinking now I need to do it on the trust as well.  

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!