topic Re: How to filter O365 API feed? in General Topics

How to filter O365 API feed?

CyberforceHero203 — Tue, 02 Jul 2019 05:25:01 GMT

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct?

create a new prototype of the IPv4Generic processor

create infilters for that

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - o365_category == 'Allow'
    name: accept o365_categoryAllow
-   actions:
    - accept
    conditions:
    - o365_category == 'Optimize'
    name: accept o365_categoryOptimize
-   actions:
    - drop
    name: drop all

create a processor node using the previously selfmade prototype
set as input the o365 miner
create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

Re: How to filter O365 API feed?

lmori — Wed, 03 Jul 2019 10:46:15 GMT

Perfect! @CyberforceHero203 just tested your filters and they work as expected.

Luigi

Re: How to filter O365 API feed?

CyberforceHero203 — Fri, 05 Jul 2019 20:52:31 GMT

Hi Luigi

Thanks for the fast reply.
It looks like it works, but if I compare the output node (finally listed indicators after my filter) with the json file which is hopefully the correct source of the miner o365-api.wordwide-any (https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a7), then it hasn't the same amount of IP's (indicators).
If you modify the filter for the category "Optimize" only, then I get these 6 indicators at the output node:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

But when I check the json file, there are more indicators listed:
104.146.128.0/17
13.107.128.0/22
13.107.136.0/22
13.107.18.10/31
13.107.6.152/31
13.107.64.0/18
131.253.33.215/32
132.245.0.0/16
134.170.200.0/21
150.171.32.0/22
150.171.40.0/22
191.234.140.0/22
204.79.197.215/32
23.103.160.0/20
40.104.0.0/15
40.108.128.0/17
40.96.0.0/13
52.104.0.0/14
52.112.0.0/14
52.96.0.0/14

Do you have any explanation for that? What have I done wrong? Is it not the same source or is the handling of the processor not correct?

Another interesting thing is that. When I don't add a parameter to the output feed, then it looks like this:
104.146.128.0-104.146.255.255
13.107.136.0-13.107.139.255
150.171.40.0-150.171.43.255
40.108.128.0-40.108.255.255
52.104.0.0-52.107.255.255

And when I add the parameter "?tr=1", then it looks like this:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

Means with the CIDR notation an aditional indicator is listed (134.170.200.0/21), I have no idea why. How about you?

Best Regards
Markus

Re: How to filter O365 API feed?

lmori — Mon, 08 Jul 2019 13:39:54 GMT

Hi @CyberforceHero203,

I think I know the problem. The same CIDRs are represented multiple times in the JSON with different categories.

Let me work on an improvement for this and for @gejack request.

Luigi

Re: How to filter O365 API feed?

CyberforceHero203 — Mon, 08 Jul 2019 14:47:19 GMT

Hi Luigi

Thanks for the reply, I'm looking forward to reading from you soon 🙂

Many thanks

Markus

Re: How to filter O365 API feed?

ttsws — Wed, 10 Jul 2019 19:32:53 GMT

Hi Luigi,

I am trying to accomplish something similar.Additionally: what's the easiest way to have the miner submit the tenantName parameter to the web service?

Kind regards,

Wolfram

Re: How to filter O365 API feed?

CyberforceHero203 — Wed, 24 Jul 2019 05:35:18 GMT

Hi Luigi

Any news from your side?

Thanks & Regards

Markus

Re: How to filter O365 API feed?

lmori — Thu, 25 Jul 2019 13:30:16 GMT

Hi @CyberforceHero203,

I have a first draft of the improvement, need some days to test it further before releasing it.

Luigi

Re: How to filter O365 API feed?

CyberforceHero203 — Thu, 25 Jul 2019 14:05:41 GMT

Hi Luigi

Nice to hear, thanks for the status update.

Markus

Re: How to filter O365 API feed?

michaelseto — Tue, 06 Aug 2019 19:45:56 GMT

I'm watching out for this one too. Looking forward to a release with this iteration!

Re: How to filter O365 API feed?

lmori — Mon, 12 Aug 2019 14:54:08 GMT

Just merged the code: https://github.com/PaloAltoNetworks/minemeld-core/pull/340

It will be there in the next release (if you are not using the develop branch now)

Re: How to filter O365 API feed?

CyberforceHero203 — Mon, 12 Aug 2019 15:08:03 GMT

Hi Luigi

Great news! Do you know the release date of the next stable version which contains your new code?

Cheers Markus

Re: How to filter O365 API feed?

CyberforceHero203 — Fri, 06 Sep 2019 09:36:05 GMT

Hi Luigi

Is the stable release already available with the improvment of the filter?

Cheers Markus

Re: How to filter O365 API feed?

lmori — Tue, 17 Sep 2019 13:21:20 GMT

@CyberforceHero203 just released version 0.9.64 with the improved Miners. It adds new attributes terminating with _list that include all the value of that attribute in the different endpoints. You can use them with the filters to reliably detect specific ids, categories, required, etc.... Example:

{
    "confidence": 100,
    "first_seen": 1565616931749,
    "last_seen": 1565616931749,
    "o365_category": "Allow",
    "o365_category_list": [
        "optimize",
        "allow"
    ],
    "o365_expressRoute": true,
    "o365_expressRoute_list": [
        "true"
    ],
    "o365_id": 6,
    "o365_id_list": [
        "1",
        "2",
        "5",
        "6"
    ],
    "o365_notes": "Exchange Online POP3 migration",
    "o365_notes_list": [
        "exchange online imap4 migration",
        "exchange online pop3 migration"
    ],
    "o365_required": false,
    "o365_required_list": [
        "false",
        "true"
    ],
    "o365_serviceArea": "Exchange",
    "o365_serviceArea_list": [
        "exchange"
    ],
    "o365_tcpPorts": "995",
    "o365_tcpPorts_list": [
        "995",
        "587",
        "143",
        "993",
        "443",
        "80"
    ],
    "o365_udpPorts_list": [],
    "share_level": "green",
    "sources": [
        "worldwide-any"
    ],
    "type": "IPv6"
}

Re: How to filter O365 API feed?

CyberforceHero203 — Tue, 17 Sep 2019 15:22:44 GMT

Hi Luigi

Great, we will test it and let you know if everything works as expected.

Regards Markus