Minemeld Ageout Policys and Withdraw

[DSHDAlex]

Hello,

Im having several issues and questions about what the best practices would be for surronding ageout policys.

• Is it better to add an ageout policy to the Miners, Aggregators, or Outputs?
• If I use the following Ageout policy, if a feed sends an IP right after the age-out occurs, will the first_seen time start over?  
  • age_out:
        default: first_seen+20d
        interval: 600
        sudden_death: true
• I have a TAXII feed that is currently ignoring all withdrawal requests, but I cannot figure out why it is doing so.  Its using the base minemeld.ft.taxii.Datafeed class, and all the miners prior to it have the same ageout policy as the one above.  

[Vorskla]

@lmori 

Having issues with age-out policy on miners.

Any answers for the questions above? Anything new? 

Please, advise and thank you very much for your time!

[Vorskla]

Hello,

Still going through configurations via trial-and-error approach as the indicator numbers are not looking correct at all. 

Reviewing configuration for Output of class minemeld.ft.taxii.DataFeed

Added the following config:

infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- type == 'IPv4'
name: accept IPv4
- actions:
- drop
name: drop all

Would it be helpful to add "store_value: true" and how?

Is there anything that may be helpful to add to an OUTPUT configuration to make sure the node (somehow) does not keep accumulating indicators after every update and ignores indicators that have already been seen/aged-out ?

Any and all assistance or feedback is very much appreciated. 

[Vorskla]

@lmori  
@xhoms 
Hello,

There are still unanswered questions regarding aging of indicators, as well as minemeld working output configuration.

• Can Minemeld function with “first_seen+2d” even if the indicator is still present in the feed?
• If “first_seen+2d” is accepted, and the miner pulls the feed again with the indicator still present what happens to the indicator?

We are also trying to understand behaviors showing in our Minemeld instance such as:

                Miner node #1 has 7413 indicators

                Miner node #2 has 783 indicators

                Processor, with Miner node #1 and Miner node #2 as input, has 8196 indicators

                Output (minemeld.ft.redis.RedisSet) has 7413 indicators

                Same processor with different Output (minemeld.ft.taxii.DataFeed) and it currently has 587479 indicators – this number keeps on growing as indicators just keep getting added on until the next service restart.

Any advice, guides or hints are much appreciated and thank you very much for your time and assistance.