Minemeld Error After Period

Reply
L3 Networker

Re: Minemeld Error After Period

Right, there are two different problems. I will run the command before next reboot. 

 

I dont erase any files, really strange

L4 Transporter

Re: Minemeld Error After Period

Hi - sorry for delay.  While arranging to get the file off I noted that it was flooding with these errors:-


Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/baseserver.py", line 140, in _do_read
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/server.py", line 93, in do_read
error: [Errno 24] Too many open files
<StreamServer at 0x7fbffaa0cd90 fileno=5 address=127.0.0.1:5000 handle=<functools.partial object at 0x7fc001c0e8e8>> failed with error

 

I restarted and they stopped, so may be a good indicator?

 

Rgds

L3 Networker

Re: Minemeld Error After Period

 

I can see that number of open files is bigger than max on mine Ubuntu too..I think it can be easily increased 

 

minemeld@minemeld:/opt/minemeld/prototypes/current$ lsof | wc -l
8087
minemeld@minemeld:/opt/minemeld/prototypes/current$ ulimit -a | grep open
open files (-n) 1024
L7 Applicator

Re: Minemeld Error After Period

Hi @apackard,

thanks, that is really helpful. I checked the logs of the engine and everything was normal except for an issue with reaching ransomwaretracker. 

Before increasing the number of opened files, I would like to understand if there is a leak of file descriptors, if you run 

$ sudo ps -aef | grep gunicorn

You will find 2 processes. Could you dump the open files with "lsof -p <pid>" for each process and check if most of them are session to redis (port 6379) or rabbitmq (port 5672) ?

 

Do you have many firewalls/devices retrieving feeds from MM ?

 

Thanks,

luigi

L4 Transporter

Re: Minemeld Error After Period

Will do.

 

In terms of the question:-

 

We currently have an IP block list provided by a 3rd party.  I have some custom PS scripts that I currently run that downloads this, produces DIFF reports, does some mangling and outputs as a file for serving up on an internal web server for our Internet facing firewalls (about a dozen).

 

I'm looking to replace this with MineMeld so in future it will be supporting at least 10 devices; but until we can work out why it keeps stopping we can't proceed - so right now there isn't actually any client devices etc.

 

I'm also hoping to use some dynamic behaviouir to get round some limitations in your dynamic blocklist max sizes and block-ip duration.  As we can only serve up ~1,200 IP's (out of the 50k plus in the 3rd party IP list), and as we can only block an IP for 1 hour with THREAT block-ip action, I have a SIEM that triggers a script if it sees any of the the "non-served" IP's attacking us, or if it sees repeated block-ip actions from a common source. 

 

This will poke an offending IP to a smaller 'active' attackers list that we can use for a dynamic blocklist that will have a lifetime of a month (ex.), once that functionality is in place we may serve up to our full estate of PA's, with is over 30.

L4 Transporter

Re: Minemeld Error After Period

Here you go.

 

Hopefully this is even better - we have 2 instances of MM running (we're looking to "make" a HA pair).  I have done this for both, the first "MM1" is working, the second "MM2" is currently down,so we can see the difference.

 

<Added as ZIP as text was too long for a post>.

 

Rgds

 

 

L3 Networker

Re: Minemeld Error After Period

Attached is output of '$ top -b -n 1 -o %MEM' after crash. Also 

minemeld@minemeld:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             2010792        4   2010788   1% /dev
tmpfs             404472      716    403756   1% /run
/dev/dm-0       31613844 27997008   1987860  94% /
none                   4        0         4   0% /sys/fs/cgroup
none                5120        0      5120   0% /run/lock
none             2022344        0   2022344   0% /run/shm
none              102400        0    102400   0% /run/user
/dev/sda1         240972    40631    187900  18% /boot
minemeld@minemeld:~$ free
             total       used       free     shared    buffers     cached
Mem:       4044688    3761068     283620         52     109372    1190780
-/+ buffers/cache:    2460916    1583772
Swap:      1048572     514636     533936
L7 Applicator

Re: Minemeld Error After Period

Hi @apackard,

super useful indeed. 0.9.30 has just been released and it contains a fix for a socket leak in the API process involving session to redis. That seems exactly the issue you are facing.

Could you try upgrading your instances ?

 

Thanks,

lmori

L4 Transporter

Re: Minemeld Error After Period

Excellent.

 

Using APT we're showing:-

 

minemeld/stable 0.9.7-8 amd64 [upgradable from: 0.9.7-6]

as the latest version, should I use APT or manually install?

L4 Transporter

Re: Minemeld Error After Period

..tried it just incase and looks good for version.  I'll run over the weekend to soak test, many thanks.

 

2016-12-02 15:36:30,296 INFO:0.9.7 Package minemeld-engine current version set to 0.9.30
2016-12-02 15:36:30,299 INFO:0.9.7 Package minemeld-webui current version set to 0.9.30
2016-12-02 15:36:30,301 INFO:0.9.7 Package minemeld-prototypes current version set to 0.9.30

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!