MineMeld - how to prevent age out of DShield in a TAXII output DataFeed

[EdwinD]

Here is the basic setup that I'm having trouble with:

Miner:

dshield_blocklist:
  output: true
  prototype: dshield.block

Aggregator:

  aggregator_dshield:
    inputs:
      - dshield_blocklist
    output: true
    prototype: minemeldlocal.aggregator_dshield

Output Node:

  taxiiDataFeedDshield:
    inputs:
      - aggregator_dshield
    output: true
    prototype: stdlib.taxiiDataFeed

Sample Indicator Log:

TIMESTAMP    8/1/2018 09:21:54 -0600 #14158
SOURCE         taxiiDataFeedDshield
TYPE               TRACE / ACCEPT_UPDATE
SENDER          aggregator_dshield
INDICATOR     158.85.81.0-158.85.81.255

{
    "direction": "inbound",
    "dshield_name": "SOFTLAYER - SoftLayer Technologies Inc.,",
    "confidence": 100,
    "share_level": "green",
    "sources": [
        "dshield.block"
    ],
    "dshield_country": "US",
    "dshield_nattacks": 1754,
    "first_seen": 1515252203149,
    "dshield_email": "abuse@softlayer.com",
    "type": "IPv4",
    "last_seen": 1515252203149
}

Per MineMeld both the miner and the taxii data feed currently contain 1006 indicators, which seems accurate.  The MineMeld taxiiDataFeedDshield is consumed by a taxii client, ProofPoint Threat Response 3.4.1.   If I recreate this entire feed within both MineMeld and ProofPoint, I am able to feed indicators into ProofPoint.  However, after a period of time the indicators are removed though they still exist in the DShield blocklist.

I have attempted to make use of the ETOpen.blockIPs miner, the dshield.block miner, the hailataxii miner as well as mining the hailataxii feed directly in ProofPoint.   In each case, I see indicators in the miner and at first in the aggregator and/or output feed, but they always age out somewhere, leaving the output feed and/or aggregator with 0 indicators.

What am I doing wrong?

Thank you

[xhoms]

Hi @EdwinD ,

you comment that, over time, the aggregation and output nodes loses all their indicators. What about the miner? Does it still report the full indicator table? If that's the case then you could have missconfigured an aging out policy in the aggregator node.

[EdwinD]

In an attempt to understand MineMeld better, I have made multiple attempts at setting up a simple DShield configuration.   At some point or another I have had a configuration where indicators age out from either the miner, the aggregator or the output node.   I believe it was when I used the Hail a TAXII miner the indicators eventually age out of the miner itself.

Before I opened this discussion I spent a lot of time in this forum, in the KB articles as well as looking at the discussions at the source repo. I discovered that I want to use the an age_out default of null and sudden_death of true:

age_out:
    default: null
    sudden_death: true
    interval: 300

With my currently configuration I have the entire DShield list in my DShield specific miner and aggregator.    For testing, I have a simple html text output node using this aggregator.  The indicators are currently all listed in this output node.  The output is simple IP ranges, formatted like the DShield feed located here: http://panwdbl.appspot.com/   I have MineMeld TAXII output node with this same aggregator as the input.   This TAXII node is feeding into ProofPoint TRAP (Threat Response Auto Pull).   ProofPoint lists 0 indicators in this TAXII feed.    For what it is worth, I have several other MineMeld output nodes which are working.  The difference is that the majority of these originate from MineMeld TAXII miners mining STIX from other TAXII feeds.  Unlike the raw DShield text block list miner, these other miners of STIX data have timestamps which are correctly updating at each poll.      

In prior configurations I would see the DShield indicators age out after almost exactly an hour, almost exactly after 24 hours or 30 days.   I believe part of this is explained based on differences between having the configuration setup as production versus experimental.   I believe another reason for this is because I previously had age out set to first seen + 30 days.   I believe my current problem is that the TAXII node is creating artificial first seen, last seen and age out timestamps based upon when I first setup this instance of DShield within MineMeld.  So while the indicators are in the MineMeld TAXII node, these dates are telling ProofPoint that these indicators are aged out.   I believe what I need is a way to update the last seen and age out timestamps within the miner itself.   I specify the miner because I desire a more complex setup.  If I want several  aggregators making use of the DShield miner as an input then I need the miner to be resolving this issue.   I think..  I'm not entirely sure my analysis is accurate.

Any and all help is appriciated.   Thank you.