topic Re: AutoFocus Miner Thought in General Topics

AutoFocus Miner Thought

hallerr — Wed, 22 Mar 2017 16:02:25 GMT

Getting the minemeld engine up running the AutoFocus/TAXII redesign today, one thing I noticed while trying to create better and more targeted AutoFocus miners is that I cant create a list that maintains a specific “age”

Where many lists will allow an operator to call out the last 7 days or last 60 days of events, or being no older than that, AutoFocus only has the ability to code it for a range via JSON. Perhaps it would be a good feature to have them allow the database to process a JSON that’s a constantly no older than x? Would cut down on over producing data just to get cut back down inside minemeld.

I know it will let me find a range, but the range wont automatically follow along at a specific "Time to Live" of sorts. Might be a good idea?

Re: AutoFocus Miner Thought

lmori — Fri, 24 Mar 2017 08:33:02 GMT

Hi @hallerr,

I think you can achieve that by defining a miner based on an AF query with a specific start point and use the age_out parameter inside the Miner.

Example, with this query you can retrieve indicators of malware found in the Finance industry starting 30 days ago:

{
 "children": [
 {
 "field": "sample.malware",
 "operator": "is",
 "value": 1
 },
 {
 "field": "session.device_industry",
 "operator": "is",
 "value": "Finance"
 },
 {
 "field": "sample.create_date",
 "operator": "is after",
 "value": [
 "2017-02-23T00:00:00",
 "2017-02-23T00:00:00"
 ]
 }
 ],
 "operator": "all"
}

When you create a Miner based on this query, it will first retrieve all the indicators matching this query (last 30 days worth of indicators) and then it start polling for new samples every 60 minutes. You can then use the age_out parameters inside the prototype to decide the maximum age of indicators to be stored in MineMeld. Example: after an indicator has been retrieved, you can decide to keep it in the feed for 60 days.

Re: AutoFocus Miner Thought

hallerr — Mon, 27 Mar 2017 11:58:45 GMT

Similar to what I was thinking but the dates would then be static correct? Im more thinking of a sliding value that is always 30 days old, and maybe thats what it does but it asks for a specific date?

Re: AutoFocus Miner Thought

lmori — Mon, 27 Mar 2017 14:11:21 GMT

Hi @hallerr,

only the date in the query is static and set the initial start of the sliding window. After that, every hour, the sliding window is pushed ahead:

1 - At time 0 a query is performed to retrieve the last 30 days worth of indicators using a query with a static date (time 0 - 30 days)

2 - After that every hour, a new query is performed to retrieve the new indicators that have been added after the last query

3 - and every hour, indicators older than 30 days are aged out and removed from the feed/graph

You can achieve this today by tuning the config settings of the AF Miner. 30 days could also be an arbitrary time range.