topic Re: Dataplane under severe load - Log entries in General Topics

Dataplane under severe load - Log entries

Demast — Thu, 18 Jan 2018 16:34:19 GMT

I see occasional "Dataplane under severe load" log entries. It is now occurring most days, sometimes a few times a day. Our monitoring system never shows the CPU average over 30% so whenever it happens it is apparently very brief. I also have never noticed any particular issues that correspond with these events. The events usually occur far from peak traffic periods.

I am curious if others experience this in the same manner and what to think of it. Also curious as to what might be causing it. My main concern is that if something is causing it to spike, it might be possible for a prolonged spike which would likely cause an issue.

Re: Dataplane under severe load - Log entries

BPry — Thu, 18 Jan 2018 16:47:49 GMT

Are you sure your monitoring system is looking at the dataplane cpu and not the management cpu? That would be the first thing that I would verify. The 'show running resource-monitor' command could come in handy here if you can get the alerts and take a look at the last 60 minutes and see what your utilization percentage is.

Re: Dataplane under severe load - Log entries

Demast — Thu, 18 Jan 2018 17:35:19 GMT

I am not 100% certain what it is monitoring in regards to CPU, but it does show 2 CPU entries. One does seem to correspond to the management plane and one to the data plane. I don't expect it would show any spikes since they are super short it would not likley witness them when polling the current usage.

Whenever I log in to the PA management page, I check the dataplane CPU on the dashboard page and it also is always below 30%. Whatever is happening is likely very brief. Here are the last seven days:

Resource monitoring sampling data (per day):

CPU load (%) during last 7 days:
core    0       1       2       3       4       5       6       7
     avg max avg max avg max avg max avg max avg max avg max avg max
       *   *   7 84   8 84 10 85 10 84 14 93 14 93 14 93
       *   *   8 100 10 100 12 100 12 100 17 100 17 100 16 100
       *   *   2 99   4 100   3 99   3 99   4 99   4 100   4 99
       *   *   1 89   3 90   2 89   2 89   2 89   2 89   2 89
       *   *   1 87   3 87   2 87   2 87   3 87   3 87   3 87
       *   *   2 94   4 94   3 95   3 94   5 94   5 94   5 94
       *   *   3 99   5 99   6 99   6 99   8 99   8 99   8 99
core    8       9      10      11
     avg max avg max avg max avg max
      14 93 14 93 14 93   8 84
      16 100 16 100 16 100   9 100
       4 99   4 99   4 99   2 99
       2 89   2 89   2 89   1 89
       3 87   3 87   3 87   1 87
       5 94   4 94   4 94   2 94
       8 99   8 99   8 99   4 99

Re: Dataplane under severe load - Log entries

BPry — Thu, 18 Jan 2018 17:49:04 GMT

@Demast,

Your average load is really low, but something is definately spiking the CPU to 100% on a daily basis looking at the max.

What I would do is actually write a small powershell or bash script that simply makes API calls on an hourly basis to record the 'show running resource-monitor minute' status to a file. Once you have a few examples of what's actually going on we can start to see what exactly is causing it and start working on the why.