celery-worker 100% cpu usage

jtrevaskis — Fri, 16 Nov 2018 00:13:27 GMT

I've had 100% CPU usage from the celery-worker processes for a few weeks now, I spent some time trying to resolve this but without resolution.

I'm using a standard Ubuntu 14 appliance build.

The issue is a looping of emerging threats downloading, rendering, repeat. I don't have the db permission issue some other users have reported, where the rendering is unsuccessful because the celery-worker doesn't have permission to the DB. The rendering is successful, unfortunately it just repeats as soon as it successfully renders. I have 3 workers (3 CPUs) and all 3 workers consistently loop like this.

[2018-11-16 10:38:08,510: WARNING/Worker-3] Imported 15500 rules so far...
[2018-11-16 10:38:09,550: WARNING/Worker-3] Imported 16000 rules so far...
[2018-11-16 10:38:10,567: WARNING/Worker-3] Imported 16500 rules so far...
[2018-11-16 10:38:11,592: WARNING/Worker-3] Imported 17000 rules so far...
[2018-11-16 10:38:12,651: WARNING/Worker-3] Imported 17500 rules so far...
[2018-11-16 10:38:13,664: WARNING/Worker-3] Imported 18000 rules so far...
[2018-11-16 10:38:14,691: WARNING/Worker-3] Imported 18500 rules so far...
[2018-11-16 10:38:15,707: WARNING/Worker-3] Imported 19000 rules so far...
[2018-11-16 10:38:16,043: WARNING/Worker-3] Finished Importing 19142 rules. Committing data
[2018-11-16 10:38:16,044: INFO/Worker-3] Rendering rules.

==> mhn.log <==
2018-11-16 10:33:50,193 - /opt/mhn/server/mhn/tasks/rules.py - Fetching sources from 1 sources.
2018-11-16 10:33:50,195 - /opt/mhn/server/mhn/tasks/rules.py - Downloading from "http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz".
2018-11-16 10:34:16,022 - /opt/mhn/server/mhn/tasks/rules.py - Bulk importing 19142 rules.
2018-11-16 10:35:08,829 - /opt/mhn/server/mhn/tasks/rules.py - Rendering rules.
2018-11-16 10:37:10,118 - /opt/mhn/server/mhn/tasks/rules.py - Fetching sources from 1 sources.
2018-11-16 10:37:10,120 - /opt/mhn/server/mhn/tasks/rules.py - Downloading from "http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz".
2018-11-16 10:37:36,014 - /opt/mhn/server/mhn/tasks/rules.py - Bulk importing 19142 rules.
2018-11-16 10:38:16,044 - /opt/mhn/server/mhn/tasks/rules.py - Rendering rules.

I've tried clearing out the celery queue, this also didn't seem to help. I've also tried the normal restarting and stopping the celery worker processes, however as soon as celery-beat is restarted, this looping begins again.

Thanks for any help or suggestions, I'm stuck

Re: celery-worker 100% cpu usage

lmori — Tue, 20 Nov 2018 13:15:19 GMT

Hi @jtrevaskis,

MineMeld is not using celery. I guess you have installed MHN on your server and that is giving high CPU, but that's not related to MineMeld.

Luigi

Re: celery-worker 100% cpu usage

jtrevaskis — Tue, 20 Nov 2018 22:12:24 GMT

oh of course your right, its MHN

topic celery-worker 100% cpu usage in General Topics

celery-worker 100% cpu usage

Re: celery-worker 100% cpu usage

Re: celery-worker 100% cpu usage