Indicator Duplication in Output

Reply
L4 Transporter

Indicator Duplication in Output

Hi.

 

We are having an issue where we are seeing duplicated indicators in output feeds.  This is a problem for us as the feeds are fed into a SIEM as a lookup table, and when there are duplicates it causes a import failure.

 

The duplicates seems to be associated with the additional field function (input feeds, confidence etc) as we're using the 'withVaule' outputs i.e. we'll get entries like:-

 

1.2.3.4,"feed a"

1.2.3.4,"feed_a,feed_b"

 

As far as we can tell the issue is temporal in that after a period of time (we ingest the data on a regular schedule) the duplicate entries are removed.  Our best guess is that we're requesting the data while an internal refresh is ongoing where an updated indicator entry is added before the old one is removed.  Is this correct and is there any way to prevent this?

 

As far as we can tell there is almost always a duplicate everytime we pull the data so it seems to be an ongoing issue.

 

Thanks

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!