Minemeld Error After Period

L4 Transporter

Minemeld Error After Period

We've installed MM on Ubuntu 14.04 and everything starts and seems to work OK initially.

 

However, after a period of time it seems tro crash.  Not really sure how log, but as an example I booted yesterday used if fine for an hour or so, and this morning it had failed. 

 

A typical error (top right in red box) would be ERROR RETRIEVING MINEMELD CONFIG: Internal Server Error. - see screenshot attachment.

 

If I restart the minemeld service everything starts and all is good again for a period of time.  Nothing jumps out in the logs - is there any advice you can give on things to check?

 

Thanks

L3 Networker

Re: Minemeld Error After Period

I have same problem, but my minemeld on Ubuntu 14.05 is running syslog miner/analyzer with significant number  of logs per second received from firewall. It crashed every day, after about 20-30h.  Luigi advised to add CPU, I have now 4x4 cores (4g ram) . It's up and running since 18h , will see...

L4 Transporter

Re: Minemeld Error After Period

Thanks - will look into it.

 

Our deployment is fairly light - 2GB, 2vCPU - but the only processing we're doing over the default config is 2 new IP sources of ~70k addresses, so no in-line syslog processing.

 

Cheers

L7 Applicator

Re: Minemeld Error After Period

Hi @apackard,

70k addresses are a really low volume for MineMeld. Would you mind sending me the minemeld-engine.log file from /opt/minemeld/log ? My email address is lmori@paloaltonetworks.com

 

Additional things:

- could you check memory and disk of the instance to see if they are exhausted ?

- are you using one or more taxii data feed output nodes ? those are memory hungry, next release will cut memory usage of taxii data feeds by more than 75%.

 

Thanks,

luigi

L4 Transporter

Re: Minemeld Error After Period

lmori,

 

I've uploaded a couple of screenshots to show the current setup:-

 

Resource_Use =>Triggered a reload of the largest IP list, showing the OS level stats (htop) and MM UI reported stats.  Probably a little disingenuous as CPU on the OS hits 100% but only for a few seconds (I missed it with the screenshot), and I suspect that the refresh period on the MM UI means it lags a little.

 

Nodes => Our nodes: we've created 2 new inputs, 1 aggregator and 1 output, plus the default ones.  The inputs are based on the minemeld.ft.http.HttpFT prototype

 

Flows => Connections

 

Will attach the log to our another message as looks like 3 is max...

L4 Transporter

Re: Minemeld Error After Period

Log file (replaced any sensitive names\IP's with fake strings)

L7 Applicator

Re: Minemeld Error After Period

Hi @apackard,

the volume of indicators I see from your screenshot should be handled pretty well by MineMeld with those memory and CPU resources. Would you mind uploading also the /opt/minemeld/log/minemeld-web.log file ?

If you prefer you can send it directly to me at lmori@palo...

 

Thanks!

luigi

L3 Networker

Re: Minemeld Error After Period

Mine crashed again but I monitored cpu and it was pretty low. Most probably I run out of disk space (see attached telegraf metrics).  Should I rotate rsyslog more frequently that default ? 

L3 Networker

Re: Minemeld Error After Period

See attached to see what happens after reboot (about 8.10 am). I have disk and memory freed, and server is up and running again. Practically I have to schedule daily cron reboot of mm server

L7 Applicator

Re: Minemeld Error After Period

Hi @niuk,

do you just reboot the instance or do something more ? 

Could you run this command before reboot to check which process is using most of the memory ?

$ top -b -n 1 -o %MEM

 

About the disk, are you erasing files before reboot ? I am asking because it's strange that a reboot alone could free space from disk.

 

apackard problem should be different, his instance is handling a pretty low volume of indicators.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!