Hi.
We are having an issue where we are seeing duplicated indicators in output feeds. This is a problem for us as the feeds are fed into a SIEM as a lookup table, and when there are duplicates it causes a import failure.
The duplicates seems to be associated with the additional field function (input feeds, confidence etc) as we're using the 'withVaule' outputs i.e. we'll get entries like:-
1.2.3.4,"feed a"
1.2.3.4,"feed_a,feed_b"
As far as we can tell the issue is temporal in that after a period of time (we ingest the data on a regular schedule) the duplicate entries are removed. Our best guess is that we're requesting the data while an internal refresh is ongoing where an updated indicator entry is added before the old one is removed. Is this correct and is there any way to prevent this?
As far as we can tell there is almost always a duplicate everytime we pull the data so it seems to be an ongoing issue.
Thanks
... View more