TAXII output deduplication problem

[KVasiliy]

Hello!

Could you tell me why taxii output doesn't do data deduplication?

Is it normal behaviour or bag?

This problem is very important for us because we have huge amount of IOCs (about 450K).

TAXII output just multiply this list.

Additionally after the output toked 1000000 IOCs it just stop to accept new data until deletion of some old IOCs.

The screanshot in attachment.

[lmori · Accepted solution]

Hi @KVasiliy,

TAXII DataFeed is not "classical" feed where you can find only the active list of indicators. A TAXII DataFeed stores all the updates to the indicators. So if indicator A is updated 3 times, the TAXII DataFeed output will contain 3 indicators A with different attributes with a timestamp of when the update happened.

By default the entries in the TAXII DataFeed are removed when older than 24 hours, to store more updates you can:

- decrease the age out interval using the age_out_interval config knob in the prototype. Example:

age_out_interval: 6h

- increase the number of entries that can be store in the feed (watch the memory usage !):

max_entries: 4000000

[KVasiliy]

I can't save the feed with "max_entries" option. Is it correct parameter?

[lmori]

HI @KVasiliy,

did you specify it in a new local prototype ?

luigi

[KVasiliy]

Yes, I did.

[lmori]

Could you attach a screenshot with the error you see on the Webui ?

[KVasiliy]

I don't see any error in the WEB UI. It doesn't allow me to push "OK" button.

[lmori]

Hi @KVasiliy,

the config is not a valid YAML document, you should remove the brackets "{" & "}"

[KVasiliy]

The brackets were in the config by default. I just put in a comma and the config was accepted.

Now it's working. Is it normal behavior?

[KVasiliy]

So, I think it's normal.

Before I save the config it looks like this:

{

   age_out_interval: 6h,

   max_entries: 4000000

}

But when it was saved, it look different.

[lmori]

Hi @KVasiliy,

yes, that's normal. Configs are in YAML format, and once saved they are rendered in the default MineMeld YAML formatting convention that does not include brackets.

[KVasiliy]

Thanks!