topic Re: Cannot apply advanced filters for O365 API feeds in General Topics

Cannot apply advanced filters for O365 API feeds

benime — Mon, 01 Oct 2018 17:55:44 GMT

I'm trying to filter out unneeded/unnecessary indicators from our O365 feed, but no matter where I apply the filters I am still receiving all of the indicators.

For example, I would like to filter on only indicators available over Express Route, and in the JSON you can see that 'expressRoute' is an available field with a boolean value of either true or false, but trying to add an infilter or outfilter condition for 'o365_expressRoute' doesn't work. I just end up with 0 indicators in my output due to the drop all at the end of my condition statements.

Can someone explain to me why this is? There's even an available prototype that comes with MineMeld for filtering out 3rd Party Integrations from the O365 API feed, and even that doesn't work.

Re: Cannot apply advanced filters for O365 API feeds

xhoms — Wed, 03 Oct 2018 21:17:16 GMT

Hi @benime,

just attempted to reproduce your experience in my lab unsuccesfully. In my case, the input filter for the Express Route condition works like a charm.

This is my infilter configuration:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - o365_expressRoute == true
    name: accept o365_expressRoute
-   actions:
    - drop
    name: drop all

Re: Cannot apply advanced filters for O365 API feeds

benime — Wed, 03 Oct 2018 22:05:40 GMT

@xhoms, It's funny because I was using almost the exact same syntax/conditions you were; except I was using single quotes around the 'true' in the o365_expressRoute condition.

Once I removed those it works as expected, now.

Thanks for following up!

Re: Cannot apply advanced filters for O365 API feeds

aleksandar.astardzhiev — Wed, 31 Oct 2018 10:26:14 GMT

Hi Guys,

I have tried to accomplish something similar, but instead of filtering on expressRoute I wanted to filter on "required" JSON field.

It seems I have able to accomplish this, but I still don't understnad why you need to append "o365_" to the name of the JSON field:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - type == 'URL'
    - o365_required == true
    name: accept required URL only
-   actions:
    - drop
    name: drop all
whitelist_prefixes:
- wl

Re: Cannot apply advanced filters for O365 API feeds

xhoms — Mon, 05 Nov 2018 11:59:05 GMT

Hi @aleksandar.astardzhiev,

current O365_API miner implementation prefixes the attributes 'expressRoute', 'optionalImpact', 'serviceArea', 'tcpPorts', 'udpPorts', 'category' and 'required' with the 'o365_' string. That is the reason.

https://github.com/PaloAltoNetworks/minemeld-core/blob/dc261fe35614b87ab554b08a0e2a7962850bb23d/minemeld/ft/o365.py#L276

Re: Cannot apply advanced filters for O365 API feeds

CyberforceHero203 — Thu, 20 Jun 2019 14:55:17 GMT

Hi @xhoms,

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct?

create a new prototype of the IPv4Generic processor

create infilters for that

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - o365_category == 'Allow'
    name: accept o365_categoryAllow
-   actions:
    - accept
    conditions:
    - o365_category == 'Optimize'
    name: accept o365_categoryOptimize
-   actions:
    - drop
    name: drop all

create a processor node using the previously selfmade prototype
set as input the o365 miner
create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

Re: Cannot apply advanced filters for O365 API feeds

Potato-soup — Wed, 22 Jan 2020 11:02:34 GMT

I have created a feed of O365 URLs filtering on expressRoute = True

infilters: - actions: - accept conditions: - __method == 'withdraw' name: accept withdraws - actions: - accept conditions: - o365_expressRoute == true name: accept o365_expressRoute - actions: - drop name: drop all

All works fine... However,

Comparing the results from Minemeld with the output from a bit of PowerShell, which does the same job. Minemeld does not include the URL *.outlook.office.com.

(Invoke-WebRequest -Uri $EndpointUri | ConvertFrom-Json) | % { if ($_.expressRoute) { $_.urls } }

I can not see why.