topic Re: MineMeld - debugging miner? in General Topics

MineMeld - debugging miner?

BRosenba — Fri, 14 Jul 2017 17:37:03 GMT

Is there any way to run minemeld in a debug mode that would output the command being run when a miner queries a remote source/feed?

Re: MineMeld - debugging miner?

BRosenba — Wed, 02 Aug 2017 20:47:32 GMT

Bumping this thread in hopes that someone with more detailed knowledge, may see and be able to help.

Re: MineMeld - debugging miner?

reaper — Thu, 03 Aug 2017 09:13:35 GMT

moved this message to the MineMeld discussions 🙂

Re: MineMeld - debugging miner?

lmori — Fri, 04 Aug 2017 12:31:07 GMT

Hi @BRosenba,

which prototype/Miner class are you interested in debugging ?

Thanks,

luigi

Re: MineMeld - debugging miner?

BRosenba — Fri, 04 Aug 2017 12:35:32 GMT

The CIF miner. We're experiencing some behavior that is different than we expected with the number of indicators that persist after setting sudden_death:true, and not sure if indicators are dropping from the feed that is pulled from the source or if there is other logic in the miner that is contributing to the behavior. When running a standard cif query, the number of indicators is consistent.

Re: MineMeld - debugging miner?

lmori — Mon, 07 Aug 2017 07:30:57 GMT

Hi @BRosenba,

in the CIF Miner queries are incremental, that basically means that:

- the first query performed after you create the node is for the last 7 days worth of indicators (this can be set using the initial_days parameter in the prototype)

- after the first query, each query will grab all the indicators added to the CIF server since the last poll

If you set sudden_death to true, only the indicators retrieved with the last query will be stored - all the old ones will be expired. The default behavior of the prototype is:

            age_out:
                default: last_seen+30d
                sudden_death: false

That means that an indicator will survive for 30 days after the last time it was seen in a query.

That being said you can edit the file /opt/minemeld/supervisors/config/conf.d/minemeld-engine.conf and add "--verbose" on the command line. After that "sudo service minemeld restart". You will then be able to see the CIF query in the minemeld-engine.log file.

Re: MineMeld - debugging miner?

BRosenba — Mon, 09 Oct 2017 19:28:18 GMT

After enabling the --verbose command line, I can see the CIF parameters of --reporttime and --reporttimeend with the values being about 60 minutes apart.

Essentially, I'm trying to understand the logic of the CIF miner to figure out how to properly configure a miner to receive the appropriate # of indicators from the feeds that we have access to while working within the limitations of the external block lists for the firewall model we utilize.

How does the last_seen value get calculated? Does minemeld store the indicators after each run and set the last_seen value for each indicator and compare regularly. Thanks for your help - Bill

Re: MineMeld - debugging miner?

lmori — Mon, 16 Oct 2017 04:08:14 GMT

Hi @BRosenba,

yes, exactly. The CIF miner (as most of the Miners) have a local table where they store indicators. The indicators are refreshed after each poll, and based on the final result age out policy is applied. So basically using a last_seen+30d age out policy an indicator will expire 30 days after the last poll where it showed up.