topic Re: Minemeld Error After Period in General Topics

Minemeld Error After Period

apackard — Wed, 30 Nov 2016 12:14:00 GMT

We've installed MM on Ubuntu 14.04 and everything starts and seems to work OK initially.

However, after a period of time it seems tro crash. Not really sure how log, but as an example I booted yesterday used if fine for an hour or so, and this morning it had failed.

A typical error (top right in red box) would be ERROR RETRIEVING MINEMELD CONFIG: Internal Server Error. - see screenshot attachment.

If I restart the minemeld service everything starts and all is good again for a period of time. Nothing jumps out in the logs - is there any advice you can give on things to check?

Thanks

Re: Minemeld Error After Period

niuk — Wed, 30 Nov 2016 12:30:02 GMT

I have same problem, but my minemeld on Ubuntu 14.05 is running syslog miner/analyzer with significant number of logs per second received from firewall. It crashed every day, after about 20-30h. Luigi advised to add CPU, I have now 4x4 cores (4g ram) . It's up and running since 18h , will see...

Re: Minemeld Error After Period

apackard — Wed, 30 Nov 2016 13:19:49 GMT

Thanks - will look into it.

Our deployment is fairly light - 2GB, 2vCPU - but the only processing we're doing over the default config is 2 new IP sources of ~70k addresses, so no in-line syslog processing.

Cheers

Re: Minemeld Error After Period

lmori — Wed, 30 Nov 2016 14:16:47 GMT

Hi @apackard,

70k addresses are a really low volume for MineMeld. Would you mind sending me the minemeld-engine.log file from /opt/minemeld/log ? My email address is lmori@paloaltonetworks.com

Additional things:

- could you check memory and disk of the instance to see if they are exhausted ?

- are you using one or more taxii data feed output nodes ? those are memory hungry, next release will cut memory usage of taxii data feeds by more than 75%.

Thanks,

luigi

Re: Minemeld Error After Period

apackard — Wed, 30 Nov 2016 15:46:26 GMT

lmori,

I've uploaded a couple of screenshots to show the current setup:-

Resource_Use =>Triggered a reload of the largest IP list, showing the OS level stats (htop) and MM UI reported stats. Probably a little disingenuous as CPU on the OS hits 100% but only for a few seconds (I missed it with the screenshot), and I suspect that the refresh period on the MM UI means it lags a little.

Nodes => Our nodes: we've created 2 new inputs, 1 aggregator and 1 output, plus the default ones. The inputs are based on the minemeld.ft.http.HttpFT prototype

Flows => Connections

Will attach the log to our another message as looks like 3 is max...

Re: Minemeld Error After Period

apackard — Wed, 30 Nov 2016 19:18:43 GMT

Log file (replaced any sensitive names\IP's with fake strings)

Re: Minemeld Error After Period

lmori — Wed, 30 Nov 2016 19:46:20 GMT

Hi @apackard,

the volume of indicators I see from your screenshot should be handled pretty well by MineMeld with those memory and CPU resources. Would you mind uploading also the /opt/minemeld/log/minemeld-web.log file ?

If you prefer you can send it directly to me at lmori@palo...

Thanks!

luigi

Re: Minemeld Error After Period

niuk — Thu, 01 Dec 2016 02:24:36 GMT

Mine crashed again but I monitored cpu and it was pretty low. Most probably I run out of disk space (see attached telegraf metrics). Should I rotate rsyslog more frequently that default ?

Re: Minemeld Error After Period

niuk — Thu, 01 Dec 2016 13:20:49 GMT

See attached to see what happens after reboot (about 8.10 am). I have disk and memory freed, and server is up and running again. Practically I have to schedule daily cron reboot of mm server

Re: Minemeld Error After Period

lmori — Thu, 01 Dec 2016 14:33:08 GMT

Hi @niuk,

do you just reboot the instance or do something more ?

Could you run this command before reboot to check which process is using most of the memory ?

$ top -b -n 1 -o %MEM

About the disk, are you erasing files before reboot ? I am asking because it's strange that a reboot alone could free space from disk.

apackard problem should be different, his instance is handling a pretty low volume of indicators.

Re: Minemeld Error After Period

niuk — Thu, 01 Dec 2016 14:35:37 GMT

Right, there are two different problems. I will run the command before next reboot.

I dont erase any files, really strange

Re: Minemeld Error After Period

apackard — Fri, 02 Dec 2016 12:44:11 GMT

Hi - sorry for delay. While arranging to get the file off I noted that it was flooding with these errors:-

Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/baseserver.py", line 140, in _do_read
File "/opt/minemeld/engine/0.9.28/local/lib/python2.7/site-packages/gevent/server.py", line 93, in do_read
error: [Errno 24] Too many open files
<StreamServer at 0x7fbffaa0cd90 fileno=5 address=127.0.0.1:5000 handle=<functools.partial object at 0x7fc001c0e8e8>> failed with error

I restarted and they stopped, so may be a good indicator?

Rgds

Re: Minemeld Error After Period

niuk — Fri, 02 Dec 2016 12:53:50 GMT

I can see that number of open files is bigger than max on mine Ubuntu too..I think it can be easily increased

minemeld@minemeld:/opt/minemeld/prototypes/current$ lsof | wc -l
8087
minemeld@minemeld:/opt/minemeld/prototypes/current$ ulimit -a | grep open
open files (-n) 1024

Re: Minemeld Error After Period

lmori — Fri, 02 Dec 2016 14:08:24 GMT

Hi @apackard,

thanks, that is really helpful. I checked the logs of the engine and everything was normal except for an issue with reaching ransomwaretracker.

Before increasing the number of opened files, I would like to understand if there is a leak of file descriptors, if you run

$ sudo ps -aef | grep gunicorn

You will find 2 processes. Could you dump the open files with "lsof -p <pid>" for each process and check if most of them are session to redis (port 6379) or rabbitmq (port 5672) ?

Do you have many firewalls/devices retrieving feeds from MM ?

Thanks,

luigi

Re: Minemeld Error After Period

apackard — Fri, 02 Dec 2016 14:22:30 GMT

Will do.

In terms of the question:-

We currently have an IP block list provided by a 3rd party. I have some custom PS scripts that I currently run that downloads this, produces DIFF reports, does some mangling and outputs as a file for serving up on an internal web server for our Internet facing firewalls (about a dozen).

I'm looking to replace this with MineMeld so in future it will be supporting at least 10 devices; but until we can work out why it keeps stopping we can't proceed - so right now there isn't actually any client devices etc.

I'm also hoping to use some dynamic behaviouir to get round some limitations in your dynamic blocklist max sizes and block-ip duration. As we can only serve up ~1,200 IP's (out of the 50k plus in the 3rd party IP list), and as we can only block an IP for 1 hour with THREAT block-ip action, I have a SIEM that triggers a script if it sees any of the the "non-served" IP's attacking us, or if it sees repeated block-ip actions from a common source.

This will poke an offending IP to a smaller 'active' attackers list that we can use for a dynamic blocklist that will have a lifetime of a month (ex.), once that functionality is in place we may serve up to our full estate of PA's, with is over 30.

Re: Minemeld Error After Period

apackard — Fri, 02 Dec 2016 14:36:13 GMT

Here you go.

Hopefully this is even better - we have 2 instances of MM running (we're looking to "make" a HA pair). I have done this for both, the first "MM1" is working, the second "MM2" is currently down,so we can see the difference.

<Added as ZIP as text was too long for a post>.

Rgds

Re: Minemeld Error After Period

niuk — Fri, 02 Dec 2016 15:13:02 GMT

Attached is output of '$ top -b -n 1 -o %MEM' after crash. Also

minemeld@minemeld:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             2010792        4   2010788   1% /dev
tmpfs             404472      716    403756   1% /run
/dev/dm-0       31613844 27997008   1987860  94% /
none                   4        0         4   0% /sys/fs/cgroup
none                5120        0      5120   0% /run/lock
none             2022344        0   2022344   0% /run/shm
none              102400        0    102400   0% /run/user
/dev/sda1         240972    40631    187900  18% /boot
minemeld@minemeld:~$ free
             total       used       free     shared    buffers     cached
Mem:       4044688    3761068     283620         52     109372    1190780
-/+ buffers/cache:    2460916    1583772
Swap:      1048572     514636     533936

Re: Minemeld Error After Period

lmori — Fri, 02 Dec 2016 15:31:42 GMT

Hi @apackard,

super useful indeed. 0.9.30 has just been released and it contains a fix for a socket leak in the API process involving session to redis. That seems exactly the issue you are facing.

Could you try upgrading your instances ?

Thanks,

lmori

Re: Minemeld Error After Period

apackard — Fri, 02 Dec 2016 15:35:29 GMT

Excellent.

Using APT we're showing:-

minemeld/stable 0.9.7-8 amd64 [upgradable from: 0.9.7-6]

as the latest version, should I use APT or manually install?

Re: Minemeld Error After Period

apackard — Fri, 02 Dec 2016 15:37:26 GMT

..tried it just incase and looks good for version. I'll run over the weekend to soak test, many thanks.

2016-12-02 15:36:30,296 INFO:0.9.7 Package minemeld-engine current version set to 0.9.30
2016-12-02 15:36:30,299 INFO:0.9.7 Package minemeld-webui current version set to 0.9.30
2016-12-02 15:36:30,301 INFO:0.9.7 Package minemeld-prototypes current version set to 0.9.30