MineMeld - how to prevent age out of DShield in a TAXII output DataFeed

Showing results for 
Search instead for 
Did you mean: 

MineMeld - how to prevent age out of DShield in a TAXII output DataFeed

L3 Networker

Here is the basic setup that I'm having trouble with:




  output: true
  prototype: dshield.block




      - dshield_blocklist
    output: true
    prototype: minemeldlocal.aggregator_dshield

Output Node:



      - aggregator_dshield
    output: true
    prototype: stdlib.taxiiDataFeed



Sample Indicator Log:


TIMESTAMP    8/1/2018 09:21:54 -0600 #14158
SOURCE         taxiiDataFeedDshield
SENDER          aggregator_dshield

    "direction": "inbound",
    "dshield_name": "SOFTLAYER - SoftLayer Technologies Inc.,",
    "confidence": 100,
    "share_level": "green",
    "sources": [
    "dshield_country": "US",
    "dshield_nattacks": 1754,
    "first_seen": 1515252203149,
    "dshield_email": "abuse@softlayer.com",
    "type": "IPv4",
    "last_seen": 1515252203149

Per MineMeld both the miner and the taxii data feed currently contain 1006 indicators, which seems accurate.  The MineMeld taxiiDataFeedDshield is consumed by a taxii client, ProofPoint Threat Response 3.4.1.   If I recreate this entire feed within both MineMeld and ProofPoint, I am able to feed indicators into ProofPoint.  However, after a period of time the indicators are removed though they still exist in the DShield blocklist.


I have attempted to make use of the ETOpen.blockIPs miner, the dshield.block miner, the hailataxii miner as well as mining the hailataxii feed directly in ProofPoint.   In each case, I see indicators in the miner and at first in the aggregator and/or output feed, but they always age out somewhere, leaving the output feed and/or aggregator with 0 indicators.


What am I doing wrong?


Thank you




L5 Sessionator

Hi @EdwinD ,


you comment that, over time, the aggregation and output nodes loses all their indicators. What about the miner? Does it still report the full indicator table? If that's the case then you could have missconfigured an aging out policy in the aggregator node.

In an attempt to understand MineMeld better, I have made multiple attempts at setting up a simple DShield configuration.   At some point or another I have had a configuration where indicators age out from either the miner, the aggregator or the output node.   I believe it was when I used the Hail a TAXII miner the indicators eventually age out of the miner itself.


Before I opened this discussion I spent a lot of time in this forum, in the KB articles as well as looking at the discussions at the source repo. I discovered that I want to use the an age_out default of null and sudden_death of true:

    default: null
    sudden_death: true
    interval: 300


With my currently configuration I have the entire DShield list in my DShield specific miner and aggregator.    For testing, I have a simple html text output node using this aggregator.  The indicators are currently all listed in this output node.  The output is simple IP ranges, formatted like the DShield feed located here: http://panwdbl.appspot.com/   I have MineMeld TAXII output node with this same aggregator as the input.   This TAXII node is feeding into ProofPoint TRAP (Threat Response Auto Pull).   ProofPoint lists 0 indicators in this TAXII feed.    For what it is worth, I have several other MineMeld output nodes which are working.  The difference is that the majority of these originate from MineMeld TAXII miners mining STIX from other TAXII feeds.  Unlike the raw DShield text block list miner, these other miners of STIX data have timestamps which are correctly updating at each poll.      


In prior configurations I would see the DShield indicators age out after almost exactly an hour, almost exactly after 24 hours or 30 days.   I believe part of this is explained based on differences between having the configuration setup as production versus experimental.   I believe another reason for this is because I previously had age out set to first seen + 30 days.   I believe my current problem is that the TAXII node is creating artificial first seen, last seen and age out timestamps based upon when I first setup this instance of DShield within MineMeld.  So while the indicators are in the MineMeld TAXII node, these dates are telling ProofPoint that these indicators are aged out.   I believe what I need is a way to update the last seen and age out timestamps within the miner itself.   I specify the miner because I desire a more complex setup.  If I want several  aggregators making use of the DShield miner as an input then I need the miner to be resolving this issue.   I think..  I'm not entirely sure my analysis is accurate.


Any and all help is appriciated.   Thank you.   



Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!