TAXII output deduplication problem

L2 Linker

TAXII output deduplication problem

Hello!

Could you tell me why taxii output doesn't do data deduplication?

Is it normal behaviour or bag?

This problem is very important for us because we have huge amount of IOCs (about 450K).

TAXII output just multiply this list.

Additionally after the output toked 1000000 IOCs it just stop to accept new data until deletion of some old IOCs.

The screanshot in attachment.

L7 Applicator

Re: TAXII output deduplication problem

Hi @KVasiliy,

TAXII DataFeed is not "classical" feed where you can find only the active list of indicators. A TAXII DataFeed stores all the updates to the indicators. So if indicator A is updated 3 times, the TAXII DataFeed output will contain 3 indicators A with different attributes with a timestamp of when the update happened.

By default the entries in the TAXII DataFeed are removed when older than 24 hours, to store more updates you can:

- decrease the age out interval using the age_out_interval config knob in the prototype. Example:

age_out_interval: 6h

- increase the number of entries that can be store in the feed (watch the memory usage !):

max_entries: 4000000
L2 Linker

Re: TAXII output deduplication problem

I can't save the feed with "max_entries" option. Is it correct parameter?

L7 Applicator

Re: TAXII output deduplication problem

HI @KVasiliy,

did you specify it in a new local prototype ?

 

luigi

L2 Linker

Re: TAXII output deduplication problem

Yes, I did.

L7 Applicator

Re: TAXII output deduplication problem

Could you attach a screenshot with the error you see on the Webui ?

L2 Linker

Re: TAXII output deduplication problem

I don't see any error in the WEB UI. It doesn't allow me to push "OK" button.

 

 

L7 Applicator

Re: TAXII output deduplication problem

Hi @KVasiliy,

the config is not a valid YAML document, you should remove the brackets "{" & "}"

L2 Linker

Re: TAXII output deduplication problem

The brackets were in the config by default. I just put in a comma and the config was accepted.

Now it's working. Is it normal behavior?

L2 Linker

Re: TAXII output deduplication problem

So, I think it's normal.

Before I save the config it looks like this:

{

   age_out_interval: 6h,

   max_entries: 4000000

}

But when it was saved, it look different.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!