How to filter O365 API feed?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

How to filter O365 API feed?

L1 Bithead

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct? 

  1. create a new prototype of the IPv4Generic processor
  2. create infilters for that
    infilters:
    -   actions:
        - accept
        conditions:
        - __method == 'withdraw'
        name: accept withdraws
    -   actions:
        - accept
        conditions:
        - o365_category == 'Allow'
        name: accept o365_categoryAllow
    - actions:
    - accept
    conditions:
    - o365_category == 'Optimize'
    name: accept o365_categoryOptimize - actions: - drop name: drop all
  3. create a processor node using the previously selfmade prototype
  4. set as input the o365 miner
  5. create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

14 REPLIES 14

L7 Applicator

Perfect! @mfepan just tested your filters and they work as expected.

 

Luigi

Hi Luigi

Thanks for the fast reply.
It looks like it works, but if I compare the output node (finally listed indicators after my filter) with the json file which is hopefully the correct source of the miner o365-api.wordwide-any (https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a...), then it hasn't the same amount of IP's (indicators).
If you modify the filter for the category "Optimize" only, then I get these 6 indicators at the output node:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

But when I check the json file, there are more indicators listed:
104.146.128.0/17
13.107.128.0/22
13.107.136.0/22
13.107.18.10/31
13.107.6.152/31
13.107.64.0/18
131.253.33.215/32
132.245.0.0/16
134.170.200.0/21
150.171.32.0/22
150.171.40.0/22
191.234.140.0/22
204.79.197.215/32
23.103.160.0/20
40.104.0.0/15
40.108.128.0/17
40.96.0.0/13
52.104.0.0/14
52.112.0.0/14
52.96.0.0/14

Do you have any explanation for that? What have I done wrong? Is it not the same source or is the handling of the processor not correct?

Another interesting thing is that. When I don't add a parameter to the output feed, then it looks like this:
104.146.128.0-104.146.255.255
13.107.136.0-13.107.139.255
150.171.40.0-150.171.43.255
40.108.128.0-40.108.255.255
52.104.0.0-52.107.255.255

And when I add the parameter "?tr=1", then it looks like this:
104.146.128.0/17
13.107.136.0/22
134.170.200.0/21
150.171.40.0/22
40.108.128.0/17
52.104.0.0/14

Means with the CIDR notation an aditional indicator is listed (134.170.200.0/21), I have no idea why. How about you?


Best Regards
Markus

Hi @mfepan,

I think I know the problem. The same CIDRs are represented multiple times in the JSON with different categories.

Let me work on an improvement for this and for @gejack request.

 

Luigi

Hi Luigi

 

Thanks for the reply, I'm looking forward to reading from you soon 🙂

 

Many thanks

Markus

Hi Luigi,

 

I am trying to accomplish something similar.Additionally: what's the easiest way to have the miner submit the tenantName parameter to the web service?

 

Kind regards,

Wolfram

Hi Luigi

Any news from your side?

Thanks & Regards

Markus

Hi @mfepan,

I have a first draft of the improvement, need some days to test it further before releasing it.

 

Luigi

Hi Luigi

Nice to hear, thanks for the status update.

Markus

I'm watching out for this one too.  Looking forward to a release with this iteration!

 

 

Just merged the code: https://github.com/PaloAltoNetworks/minemeld-core/pull/340

It will be there in the next release (if you are not using the develop branch now)

Hi Luigi

 

Great news! Do you know the release date of the next stable version which contains your new code?

 

Cheers Markus

Hi Luigi

Is the stable release already available with the improvment of the filter?

Cheers Markus

@mfepan just released version 0.9.64 with the improved Miners. It adds new attributes terminating with _list that include all the value of that attribute in the different endpoints. You can use them with the filters to reliably detect specific ids, categories, required, etc.... Example:

{
    "confidence": 100,
    "first_seen": 1565616931749,
    "last_seen": 1565616931749,
    "o365_category": "Allow",
    "o365_category_list": [
        "optimize",
        "allow"
    ],
    "o365_expressRoute": true,
    "o365_expressRoute_list": [
        "true"
    ],
    "o365_id": 6,
    "o365_id_list": [
        "1",
        "2",
        "5",
        "6"
    ],
    "o365_notes": "Exchange Online POP3 migration",
    "o365_notes_list": [
        "exchange online imap4 migration",
        "exchange online pop3 migration"
    ],
    "o365_required": false,
    "o365_required_list": [
        "false",
        "true"
    ],
    "o365_serviceArea": "Exchange",
    "o365_serviceArea_list": [
        "exchange"
    ],
    "o365_tcpPorts": "995",
    "o365_tcpPorts_list": [
        "995",
        "587",
        "143",
        "993",
        "443",
        "80"
    ],
    "o365_udpPorts_list": [],
    "share_level": "green",
    "sources": [
        "worldwide-any"
    ],
    "type": "IPv6"
}

 

Hi Luigi

Great, we will test it and let you know if everything works as expected.

Regards Markus

  • 10741 Views
  • 14 replies
  • 1 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!