How to filter O365 API feed?

[mfepan]

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct? 

1. create a new prototype of the IPv4Generic processor
2. create infilters for that
   

   infilters:
   -   actions:
       - accept
       conditions:
       - __method == 'withdraw'
       name: accept withdraws
   -   actions:
       - accept
       conditions:
       - o365_category == 'Allow'
       name: accept o365_categoryAllow
   -   actions:
       - accept
       conditions:
       - o365_category == 'Optimize'
       name: accept o365_categoryOptimize
   -   actions:
       - drop
       name: drop all

3. create a processor node using the previously selfmade prototype
4. set as input the o365 miner
5. create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

[lmori]

Perfect! @mfepan just tested your filters and they work as expected.

Luigi

[mfepan]

Hi Luigi

Thanks for the fast reply.
It looks like it works, but if I compare the output node (finally listed indicators after my filter) with the json file which is hopefully the correct source of the miner o365-api.wordwide-any (https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a...), then it hasn't the same amount of IP's (indicators).
If you modify the filter for the category "Optimize" only, then I get these 6 indicators at the output node:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

But when I check the json file, there are more indicators listed:
104.146.128.0/17
13.107.128.0/22
13.107.136.0/22
13.107.18.10/31
13.107.6.152/31
13.107.64.0/18
131.253.33.215/32
132.245.0.0/16
134.170.200.0/21
150.171.32.0/22
150.171.40.0/22
191.234.140.0/22
204.79.197.215/32
23.103.160.0/20
40.104.0.0/15
40.108.128.0/17
40.96.0.0/13
52.104.0.0/14
52.112.0.0/14
52.96.0.0/14

Do you have any explanation for that? What have I done wrong? Is it not the same source or is the handling of the processor not correct?

Another interesting thing is that. When I don't add a parameter to the output feed, then it looks like this:
104.146.128.0-104.146.255.255
13.107.136.0-13.107.139.255
150.171.40.0-150.171.43.255
40.108.128.0-40.108.255.255
52.104.0.0-52.107.255.255

And when I add the parameter "?tr=1", then it looks like this:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

Means with the CIDR notation an aditional indicator is listed (134.170.200.0/21), I have no idea why. How about you?

Best Regards
Markus

[lmori]

Hi @mfepan,

I think I know the problem. The same CIDRs are represented multiple times in the JSON with different categories.

Let me work on an improvement for this and for @gejack request.

Luigi

[mfepan]

Hi Luigi

Thanks for the reply, I'm looking forward to reading from you soon 🙂

Many thanks

Markus

[ttsws]

Hi Luigi,

I am trying to accomplish something similar.Additionally: what's the easiest way to have the miner submit the tenantName parameter to the web service?

Kind regards,

Wolfram

[mfepan]

Hi Luigi

Any news from your side?

Thanks & Regards

Markus

[lmori]

Hi @mfepan,

I have a first draft of the improvement, need some days to test it further before releasing it.

Luigi

[mfepan]

Hi Luigi

Nice to hear, thanks for the status update.

Markus

[michaelseto]

I'm watching out for this one too.  Looking forward to a release with this iteration!

[lmori]

Just merged the code: https://github.com/PaloAltoNetworks/minemeld-core/pull/340

It will be there in the next release (if you are not using the develop branch now)

[mfepan]

Hi Luigi

Great news! Do you know the release date of the next stable version which contains your new code?

Cheers Markus

[mfepan]

Hi Luigi

Is the stable release already available with the improvment of the filter?

Cheers Markus

[lmori]

@mfepan just released version 0.9.64 with the improved Miners. It adds new attributes terminating with _list that include all the value of that attribute in the different endpoints. You can use them with the filters to reliably detect specific ids, categories, required, etc.... Example:

{
    "confidence": 100,
    "first_seen": 1565616931749,
    "last_seen": 1565616931749,
    "o365_category": "Allow",
    "o365_category_list": [
        "optimize",
        "allow"
    ],
    "o365_expressRoute": true,
    "o365_expressRoute_list": [
        "true"
    ],
    "o365_id": 6,
    "o365_id_list": [
        "1",
        "2",
        "5",
        "6"
    ],
    "o365_notes": "Exchange Online POP3 migration",
    "o365_notes_list": [
        "exchange online imap4 migration",
        "exchange online pop3 migration"
    ],
    "o365_required": false,
    "o365_required_list": [
        "false",
        "true"
    ],
    "o365_serviceArea": "Exchange",
    "o365_serviceArea_list": [
        "exchange"
    ],
    "o365_tcpPorts": "995",
    "o365_tcpPorts_list": [
        "995",
        "587",
        "143",
        "993",
        "443",
        "80"
    ],
    "o365_udpPorts_list": [],
    "share_level": "green",
    "sources": [
        "worldwide-any"
    ],
    "type": "IPv6"
}

[mfepan]

Hi Luigi

Great, we will test it and let you know if everything works as expected.

Regards Markus