Differences between stdlib.feedHCGreen and stdlib.taxiiDataFeed output miners

[DanWoodruff]

Hi,

I have each of my processors set up to send to two different output miners - one for the firewalls to ingest (stdlib.feedHCGreen) and one for my SIEM to ingest (stdlib.taxiiDataFeed). I'm seeing differences between what is ending up in my SIEM and what is on the list for the firewalls. I believe feedHCGreen outputs are correct while TAXII outputs are off.

Looking at the NODES screen, the number of indicators is different between the output miners. This is consistent with 6 different processors (total of 12 miners, 6 of feedHCGreen and 6 of taxiiDataFeed). Screenshots are below showing differences in counts of indicators. 

It is quite possible that I am not understanding how TAXII works or I am doing something wrong. Is it part of the protocol that indicators automatically age out after a period of time? I didn't see anything in the prototype configs that specify an age out time. The result I am trying to accomplish is both sets of outputs having the exact same data at all times.

Thanks!

Dan

[lmori · Accepted solution]

Hi Dan,

yes, there is a difference between "traditional" feeds and TAXII DataFeeds:

• in a traditional feed indicators are removed when they are aged out by Miners. These feeds basically are static, they keep indicators until they are aged out.
• a TAXII DataFeed instead is incremental, it is an orderd feed of indicators. The feed records all the updated indicators on a rolling window of 24 hours (configurable). It becomes then a responsability of the consumer to decide how to build the list of current live indicators. Example: in QRadar you can define a TTL to all the elements of the reference set.

There is a second type TAXII Data Collection called Data Set that is more similar to a traditional static feed. This is not supported yet, but we plan to add support in the short term.

Ref page 51 of http://taxiiproject.github.io/releases/1.1/TAXII_Services_Specification.pdf.

[DanWoodruff]

Luigi, this is very helpful, thank you!