topic Re: Minemeld Ageout Policys and Withdraw in General Topics

Minemeld Ageout Policys and Withdraw

DSHDAlex — Thu, 01 Aug 2019 18:40:10 GMT

Hello,

Im having several issues and questions about what the best practices would be for surronding ageout policys.

Is it better to add an ageout policy to the Miners, Aggregators, or Outputs?
If I use the following Ageout policy, if a feed sends an IP right after the age-out occurs, will the first_seen time start over?
- age_out:
  default: first_seen+20d
  interval: 600
  sudden_death: true
I have a TAXII feed that is currently ignoring all withdrawal requests, but I cannot figure out why it is doing so. Its using the base minemeld.ft.taxii.Datafeed class, and all the miners prior to it have the same ageout policy as the one above.

Re: Minemeld Ageout Policys and Withdraw

Vorskla — Wed, 20 Nov 2019 17:59:25 GMT

@lmori

Having issues with age-out policy on miners.

Any answers for the questions above? Anything new?

Please, advise and thank you very much for your time!

Re: Minemeld Ageout Policys and Withdraw

Vorskla — Wed, 27 Nov 2019 19:05:38 GMT

Hello,

Still going through configurations via trial-and-error approach as the indicator numbers are not looking correct at all.

Reviewing configuration for Output of class minemeld.ft.taxii.DataFeed

Added the following config:

infilters: - actions: - accept conditions: - __method == 'withdraw' name: accept withdraws - actions: - accept conditions: - type == 'IPv4' name: accept IPv4 - actions: - drop name: drop all

Would it be helpful to add "store_value: true" and how?

Is there anything that may be helpful to add to an OUTPUT configuration to make sure the node (somehow) does not keep accumulating indicators after every update and ignores indicators that have already been seen/aged-out ?

Any and all assistance or feedback is very much appreciated.

Re: Minemeld Ageout Policys and Withdraw

Vorskla — Fri, 20 Dec 2019 21:27:15 GMT

@lmori
@xhoms
Hello,

There are still unanswered questions regarding aging of indicators, as well as minemeld working output configuration.

Can Minemeld function with “first_seen+2d” even if the indicator is still present in the feed?
If “first_seen+2d” is accepted, and the miner pulls the feed again with the indicator still present what happens to the indicator?

We are also trying to understand behaviors showing in our Minemeld instance such as:

Miner node #1 has 7413 indicators

Miner node #2 has 783 indicators

Processor, with Miner node #1 and Miner node #2 as input, has 8196 indicators

Output (minemeld.ft.redis.RedisSet) has 7413 indicators

Same processor with different Output (minemeld.ft.taxii.DataFeed) and it currently has 587479 indicators – this number keeps on growing as indicators just keep getting added on until the next service restart.

Any advice, guides or hints are much appreciated and thank you very much for your time and assistance.