Hi - sorry for delay. While arranging to get the file off I noted that it was flooding with these errors:-
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/baseserver.py", line 140, in _do_read
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/server.py", line 93, in do_read
error: [Errno 24] Too many open files
<StreamServer at 0x7fbffaa0cd90 fileno=5 address=127.0.0.1:5000 handle=<functools.partial object at 0x7fc001c0e8e8>> failed with error
I restarted and they stopped, so may be a good indicator?
I can see that number of open files is bigger than max on mine Ubuntu too..I think it can be easily increased
minemeld@minemeld:/opt/minemeld/prototypes/current$ lsof | wc -l 8087 minemeld@minemeld:/opt/minemeld/prototypes/current$ ulimit -a | grep open open files (-n) 1024
thanks, that is really helpful. I checked the logs of the engine and everything was normal except for an issue with reaching ransomwaretracker.
Before increasing the number of opened files, I would like to understand if there is a leak of file descriptors, if you run
$ sudo ps -aef | grep gunicorn
You will find 2 processes. Could you dump the open files with "lsof -p <pid>" for each process and check if most of them are session to redis (port 6379) or rabbitmq (port 5672) ?
Do you have many firewalls/devices retrieving feeds from MM ?
In terms of the question:-
We currently have an IP block list provided by a 3rd party. I have some custom PS scripts that I currently run that downloads this, produces DIFF reports, does some mangling and outputs as a file for serving up on an internal web server for our Internet facing firewalls (about a dozen).
I'm looking to replace this with MineMeld so in future it will be supporting at least 10 devices; but until we can work out why it keeps stopping we can't proceed - so right now there isn't actually any client devices etc.
I'm also hoping to use some dynamic behaviouir to get round some limitations in your dynamic blocklist max sizes and block-ip duration. As we can only serve up ~1,200 IP's (out of the 50k plus in the 3rd party IP list), and as we can only block an IP for 1 hour with THREAT block-ip action, I have a SIEM that triggers a script if it sees any of the the "non-served" IP's attacking us, or if it sees repeated block-ip actions from a common source.
This will poke an offending IP to a smaller 'active' attackers list that we can use for a dynamic blocklist that will have a lifetime of a month (ex.), once that functionality is in place we may serve up to our full estate of PA's, with is over 30.
Here you go.
Hopefully this is even better - we have 2 instances of MM running (we're looking to "make" a HA pair). I have done this for both, the first "MM1" is working, the second "MM2" is currently down,so we can see the difference.
<Added as ZIP as text was too long for a post>.
Attached is output of '$ top -b -n 1 -o %MEM' after crash. Also
minemeld@minemeld:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 2010792 4 2010788 1% /dev tmpfs 404472 716 403756 1% /run /dev/dm-0 31613844 27997008 1987860 94% / none 4 0 4 0% /sys/fs/cgroup none 5120 0 5120 0% /run/lock none 2022344 0 2022344 0% /run/shm none 102400 0 102400 0% /run/user /dev/sda1 240972 40631 187900 18% /boot minemeld@minemeld:~$ free total used free shared buffers cached Mem: 4044688 3761068 283620 52 109372 1190780 -/+ buffers/cache: 2460916 1583772 Swap: 1048572 514636 533936
super useful indeed. 0.9.30 has just been released and it contains a fix for a socket leak in the API process involving session to redis. That seems exactly the issue you are facing.
Could you try upgrading your instances ?
Using APT we're showing:-
minemeld/stable 0.9.7-8 amd64 [upgradable from: 0.9.7-6]
as the latest version, should I use APT or manually install?
..tried it just incase and looks good for version. I'll run over the weekend to soak test, many thanks.
2016-12-02 15:36:30,296 INFO:0.9.7 Package minemeld-engine current version set to 0.9.30
2016-12-02 15:36:30,299 INFO:0.9.7 Package minemeld-webui current version set to 0.9.30
2016-12-02 15:36:30,301 INFO:0.9.7 Package minemeld-prototypes current version set to 0.9.30
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!