Output limit?

Reply
L4 Transporter

Output limit?

 

Hi,


I run minemeld (standalone) in a virtual machine with 2 cpu, 6GB RAM and 40GB hd. My config has 63 miners (mainly youtube miners and ransomware trackers), 13 aggregators and 30 output nodes. The miners start the job, but when it reach the band "85k-95k indicators", the Minemeld stops mining. The miners get the status "started" and few of them "stopped". It doesn't restart the service each x sec, it just stop mining. I know it stops mining, because one of the Youtube channels has more than 30k videos and the miner only gets few URLs, the same with the ransonware IP trackers.

 

I noticed that my Minemeld hardly aged-out or removes indicators. For example, in a typical dashboard showed in many articles, the monitor presents the number of aged-out or removed indicators as a parable (half sine). In my case, figure bellow, I have flat lines. I thought it could be something related with NTP leading Minemeld to be to out of resources with so many indicators, but the time configuration in my server is perfect.

 

Finally, some time later (there is not a specific interval) the dashbord shows 0 indicators.

 

Could someone give me any tip, advice, help?

 

Thanks in advanced.

 

L7 Applicator

Re: Output limit?

Hi @danilo.souza,

flat lines could be normal. But please could you add the minemeld-engine.log file to the thread? You can download it from System > Engine > Logs.

Please, check it before posting that it does not contain confidential information - especially credentials to access feeds.

 

Thanks,

luigi

L4 Transporter

Re: Output limit?

Hi Luigi,

thank you for the fast reply. Just to be sure, there is no way to send the log just to you, right? I have to add the file here in the forum, right?

Thank you one more time.

L7 Applicator

Re: Output limit?

Hi @danilo.souza,

sure, please send them to lmori@paloaltonetworks.com

 

Thanks,

luigi

L7 Applicator

Re: Output limit?

Hi @danilo.souza,

I checked your logs and it seems a rabbitmq malfunction. Which distribution are you using? How much memory do you have on that instance?

 

luigi

L4 Transporter

Re: Output limit?

Hi Luigi,

I am using the version 0.9.44 for CentOS. That is what I get from the engine log:

 

/opt/minemeld/log/minemeld-engine.log.6:2018-04-15T17:07:19 (2404)launcher.main INFO: Starting mm-run.py version 0.9.44.post1

 

What you mean by how much in memory in that instance? When you refers to rabbitmq, is it a bad news?

 

Thank you again.

L7 Applicator

Re: Output limit?

L4 Transporter

Re: Output limit?

Hi Luigi,

it is disabled (image attached).

Best regards.

L4 Transporter

Re: Output limit?

Hi Luigi,


Is there any other information I can provide to help identifying the problem?

 

Thank you again

L4 Transporter

Re: Output limit?

Hi,

is there anybody else with a similar case that could help in this case? It is really important.

Thanks.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!