I integrated a few custom feeds into MineMeld that have a high number of indicators. The dashboard shows 202.4K, across 21 miners. I noticed two things that I'm hoping I can work around:
Thanks for the great tool! We're excited about using it.
Solved! Go to Solution.
Disk and CPU usage heavily depend on the type of Miner being used.
For disk usage:
Could you share more details about Miners you are running and which one was using 2GB of compressed DB ?
For CPU, again it depends on the Miners. More core you add (even virtual) the better. There are also some hot spots inside MineMeld we are working on optimizing, stay tuned :-)
Thanks for some insight into the internal workings!
The miners that I am using are all the stdlib.list*Generic miners. I have a few of each IPv4, Domain, and URL. The largest feeds are in the Domain and URL categories - one is 10K indicators and another is a little more than 20K. These are the ones that were using > 2GB. For example, I restarted MineMeld yesterday and one of them is back to 863MB with 431 .ldb files.
These are populated from a proprietary feed that I unfortunately am only able to query via the CIF command line tool, so that's why I'm using the generic miners. I built a script running on cron every 3 hours to create the yml output, which is then written out to the appropriate files in /opt/minemeld/local/config/. Then when MineMeld polls, it pulls in any changes.
The only indications I have about how many are aged out come from the dashboard, and I'm looking at only a handful (< 100) added or removed at each refresh. So not a lot of churn.
I'll see about increasing from 2 to 4 core for the VM.
I am working on adding a CIF Miner to MineMeld to automate the queries. Would you have time for a quick call ? Just want to be sure I get your queries covered.
Just send me a message to email@example.com if you are ok with it.
@DanWoodruff my suggestion to lower CPU and disk usage in your case is to create a new prototype based on listIPv4Generic and increase the interval and age_out interval settings from one minute to 12 or 24 hours.
If your script calls mm-console to hup the node after creating the new list, the Miner doesn't really need to monitor the file for changes.
Thanks, I will give this a try. Are the intervals defined in seconds? Right now in my install, stdlib.listIPv4Generic has an age_out interval of 67 and a interval of 53.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!