How to filter O365 API feed?

[mfepan]

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct? 

1. create a new prototype of the IPv4Generic processor
2. create infilters for that
   

   infilters:
   -   actions:
       - accept
       conditions:
       - __method == 'withdraw'
       name: accept withdraws
   -   actions:
       - accept
       conditions:
       - o365_category == 'Allow'
       name: accept o365_categoryAllow
   -   actions:
       - accept
       conditions:
       - o365_category == 'Optimize'
       name: accept o365_categoryOptimize
   -   actions:
       - drop
       name: drop all

3. create a processor node using the previously selfmade prototype
4. set as input the o365 miner
5. create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

[lmori]

Perfect! @mfepan just tested your filters and they work as expected.

Luigi

[mfepan]

Hi Luigi

Thanks for the fast reply.
It looks like it works, but if I compare the output node (finally listed indicators after my filter) with the json file which is hopefully the correct source of the miner o365-api.wordwide-any (https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a...), then it hasn't the same amount of IP's (indicators).
If you modify the filter for the category "Optimize" only, then I get these 6 indicators at the output node:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

But when I check the json file, there are more indicators listed:
104.146.128.0/17
13.107.128.0/22
13.107.136.0/22
13.107.18.10/31
13.107.6.152/31
13.107.64.0/18
131.253.33.215/32
132.245.0.0/16
134.170.200.0/21
150.171.32.0/22
150.171.40.0/22
191.234.140.0/22
204.79.197.215/32
23.103.160.0/20
40.104.0.0/15
40.108.128.0/17
40.96.0.0/13
52.104.0.0/14
52.112.0.0/14
52.96.0.0/14

Do you have any explanation for that? What have I done wrong? Is it not the same source or is the handling of the processor not correct?

Another interesting thing is that. When I don't add a parameter to the output feed, then it looks like this:
104.146.128.0-104.146.255.255
13.107.136.0-13.107.139.255
150.171.40.0-150.171.43.255
40.108.128.0-40.108.255.255
52.104.0.0-52.107.255.255

And when I add the parameter "?tr=1", then it looks like this:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

Means with the CIDR notation an aditional indicator is listed (134.170.200.0/21), I have no idea why. How about you?

Best Regards
Markus

[lmori]

Hi @mfepan,

I think I know the problem. The same CIDRs are represented multiple times in the JSON with different categories.

Let me work on an improvement for this and for @gejack request.

Luigi

[mfepan]

Hi Luigi

Thanks for the reply, I'm looking forward to reading from you soon :-)

Many thanks

Markus

[ttsws]

Hi Luigi,

I am trying to accomplish something similar.Additionally: what's the easiest way to have the miner submit the tenantName parameter to the web service?

Kind regards,

Wolfram

[mfepan]

Hi Luigi

Any news from your side?

Thanks & Regards

Markus

[lmori]

Hi @mfepan,

I have a first draft of the improvement, need some days to test it further before releasing it.

Luigi

[mfepan]

Hi Luigi

Nice to hear, thanks for the status update.

Markus

[michaelseto]

I'm watching out for this one too.  Looking forward to a release with this iteration!