topic Re: TAXII output deduplication problem in General Topics

TAXII output deduplication problem

KVasiliy — Thu, 31 Aug 2017 08:23:35 GMT

Hello!

Could you tell me why taxii output doesn't do data deduplication?

Is it normal behaviour or bag?

This problem is very important for us because we have huge amount of IOCs (about 450K).

TAXII output just multiply this list.

Additionally after the output toked 1000000 IOCs it just stop to accept new data until deletion of some old IOCs.

The screanshot in attachment.

Re: TAXII output deduplication problem

lmori — Tue, 05 Sep 2017 12:00:11 GMT

Hi @KVasiliy,

TAXII DataFeed is not "classical" feed where you can find only the active list of indicators. A TAXII DataFeed stores all the updates to the indicators. So if indicator A is updated 3 times, the TAXII DataFeed output will contain 3 indicators A with different attributes with a timestamp of when the update happened.

By default the entries in the TAXII DataFeed are removed when older than 24 hours, to store more updates you can:

- decrease the age out interval using the age_out_interval config knob in the prototype. Example:

age_out_interval: 6h

- increase the number of entries that can be store in the feed (watch the memory usage !):

max_entries: 4000000

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 12:15:37 GMT

I can't save the feed with "max_entries" option. Is it correct parameter?

Re: TAXII output deduplication problem

lmori — Tue, 05 Sep 2017 12:29:05 GMT

HI @KVasiliy,

did you specify it in a new local prototype ?

luigi

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 12:34:30 GMT

Yes, I did.

Re: TAXII output deduplication problem

lmori — Tue, 05 Sep 2017 12:37:11 GMT

Could you attach a screenshot with the error you see on the Webui ?

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 12:44:14 GMT

I don't see any error in the WEB UI. It doesn't allow me to push "OK" button.

Re: TAXII output deduplication problem

lmori — Tue, 05 Sep 2017 12:51:08 GMT

Hi @KVasiliy,

the config is not a valid YAML document, you should remove the brackets "{" & "}"

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 12:56:50 GMT

The brackets were in the config by default. I just put in a comma and the config was accepted.

Now it's working. Is it normal behavior?

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 13:12:29 GMT

So, I think it's normal.

Before I save the config it looks like this:

{

age_out_interval: 6h,

max_entries: 4000000

}

But when it was saved, it look different.

Re: TAXII output deduplication problem

lmori — Tue, 05 Sep 2017 13:22:49 GMT

Hi @KVasiliy,

yes, that's normal. Configs are in YAML format, and once saved they are rendered in the default MineMeld YAML formatting convention that does not include brackets.

Re: TAXII output deduplication problem

KVasiliy — Tue, 05 Sep 2017 13:27:15 GMT

Thanks!