TAXII output deduplication problem

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

TAXII output deduplication problem

L2 Linker

Hello!

Could you tell me why taxii output doesn't do data deduplication?

Is it normal behaviour or bag?

This problem is very important for us because we have huge amount of IOCs (about 450K).

TAXII output just multiply this list.

Additionally after the output toked 1000000 IOCs it just stop to accept new data until deletion of some old IOCs.

The screanshot in attachment.

1 accepted solution

Accepted Solutions

L7 Applicator

Hi @KVasiliy,

TAXII DataFeed is not "classical" feed where you can find only the active list of indicators. A TAXII DataFeed stores all the updates to the indicators. So if indicator A is updated 3 times, the TAXII DataFeed output will contain 3 indicators A with different attributes with a timestamp of when the update happened.

By default the entries in the TAXII DataFeed are removed when older than 24 hours, to store more updates you can:

- decrease the age out interval using the age_out_interval config knob in the prototype. Example:

age_out_interval: 6h

- increase the number of entries that can be store in the feed (watch the memory usage !):

max_entries: 4000000

View solution in original post

11 REPLIES 11

L7 Applicator

Hi @KVasiliy,

TAXII DataFeed is not "classical" feed where you can find only the active list of indicators. A TAXII DataFeed stores all the updates to the indicators. So if indicator A is updated 3 times, the TAXII DataFeed output will contain 3 indicators A with different attributes with a timestamp of when the update happened.

By default the entries in the TAXII DataFeed are removed when older than 24 hours, to store more updates you can:

- decrease the age out interval using the age_out_interval config knob in the prototype. Example:

age_out_interval: 6h

- increase the number of entries that can be store in the feed (watch the memory usage !):

max_entries: 4000000

I can't save the feed with "max_entries" option. Is it correct parameter?

HI @KVasiliy,

did you specify it in a new local prototype ?

 

luigi

Yes, I did.

Could you attach a screenshot with the error you see on the Webui ?

I don't see any error in the WEB UI. It doesn't allow me to push "OK" button.

 

 

Hi @KVasiliy,

the config is not a valid YAML document, you should remove the brackets "{" & "}"

The brackets were in the config by default. I just put in a comma and the config was accepted.

Now it's working. Is it normal behavior?

So, I think it's normal.

Before I save the config it looks like this:

{

   age_out_interval: 6h,

   max_entries: 4000000

}

But when it was saved, it look different.

Hi @KVasiliy,

yes, that's normal. Configs are in YAML format, and once saved they are rendered in the default MineMeld YAML formatting convention that does not include brackets.

Thanks!

  • 1 accepted solution
  • 9971 Views
  • 11 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!