topic Re: Withdraw mesage source in General Topics

Withdraw mesage source

Forseti — Wed, 12 Oct 2016 08:42:38 GMT

Hi everyone,

I am currently working on connecting MineMeld with our SIEM solution. I however ran into a question.

When receiving an update message it states which sources the IOC originated from, also if there are multiple. example: (binarydefense and badips)

{"message":"{\"@indicator\":\"120.69.220.5-120.69.220.5\",\"direction\":\"inbound\",\"@origin\":\"IPv4_Aggregator\",\"type\":\"IPv4\",\"@timestamp\":\"2016-10-11T17:13:35.693563Z\",\"confidence\":50,\"share_level\":\"green\",\"sources\":[\"binarydefense.banlist\",\"badips.any_3\"],\"logstash_output_node\":\"Output-To-Logstash-5514\",\"message\":\"update\",\"@version\":1,\"first_seen\":\"2016-09-30T13:50:29.164000Z\",\"last_seen\":\"2016-09-30T13:50:34.330000Z\"}","@version":"1","@timestamp":"2016-10-11T15:13:35.693Z","host":"127.0.0.1","port":34402}

However all withdraw messages I receive do not include this field.

Question: are withdraw messages generated when the IOC is removed rom ALL sources or is a messages generated for each source individually?

Regards,

Forseti

Re: Withdraw mesage source

lmori — Wed, 12 Oct 2016 09:37:54 GMT

Hi @Forseti,

WITHDRAW messages have no body and an aggregator generates a withdraw for an indicator only when all the Miners have withdrawn the indicator.

Longer explanation:

Let's suppose you have a graph with 2 Miners M1 and M2, connected to an aggregator A, and A is connected to a single output node O. This is how WITHDRAWs work:

suppose that in the initial state both M1 and M2 have published an indicator I. M1 published I with a set of attributes AM1. M2 published I with a set of attributes AM2. Aggregator A then has published to O the indicator I with a set of attributes (AM1+AM2).
Miner M1 expires indicator I (expiration of indicators depends on the age out configuration in each single Miner, i.e. each Miner can apply a different age out policy) and sends a WITHDRAW of indicator I to aggregator A.
Aggregator A generates an UPDATE message to output O for indicator I with a set of attributes AM2, as AM1 have been removed.
Miner M2 expires indicator I (according to its own age out policy) and sends a WITHDRAW of indicator I to aggregator A.
Aggregator A at this point should remove indicator I because all the original Miners sources of I have withdrawn the indicator. Aggregator A then generates a WITHDRAW message for indicator I to output O.

Hope this helps.

Luigi

Re: Withdraw mesage source

Forseti — Wed, 12 Oct 2016 10:25:55 GMT

Perfect answer! Just what I needed to know! Thank you @Imori