Cannot apply advanced filters for O365 API feeds

[benime]

I'm trying to filter out unneeded/unnecessary indicators from our O365 feed, but no matter where I apply the filters I am still receiving all of the indicators.

For example, I would like to filter on only indicators available over Express Route, and in the JSON you can see that 'expressRoute' is an available field with a boolean value of either true or false, but trying to add an infilter or outfilter condition for 'o365_expressRoute' doesn't work. I just end up with 0 indicators in my output due to the drop all at the end of my condition statements.

Can someone explain to me why this is? There's even an available prototype that comes with MineMeld for filtering out 3rd Party Integrations from the O365 API feed, and even that doesn't work.

[xhoms · Accepted solution]

Hi @benime,

just attempted to reproduce your experience in my lab unsuccesfully. In my case, the input filter for the Express Route condition works like a charm.

This is my infilter configuration:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - o365_expressRoute == true
    name: accept o365_expressRoute
-   actions:
    - drop
    name: drop all

[benime]

@xhoms, It's funny because I was using almost the exact same syntax/conditions you were; except I was using single quotes around the 'true' in the o365_expressRoute condition.

Once I removed those it works as expected, now.

Thanks for following up!

[aleksandar.astardzhiev]

Hi Guys,

I have tried to accomplish something similar, but instead of filtering on expressRoute I wanted to filter on "required" JSON field.

It seems I have able to accomplish this, but I still don't understnad why you need to append "o365_" to the name of the JSON field:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - type == 'URL'
    - o365_required == true
    name: accept required URL only
-   actions:
    - drop
    name: drop all
whitelist_prefixes:
- wl

[xhoms]

Hi @aleksandar.astardzhiev,

current O365_API miner implementation prefixes the attributes 'expressRoute', 'optionalImpact', 'serviceArea', 'tcpPorts', 'udpPorts', 'category' and 'required' with the 'o365_' string. That is the reason.

https://github.com/PaloAltoNetworks/minemeld-core/blob/dc261fe35614b87ab554b08a0e2a7962850bb23d/mine...

[mfepan]

Hi @xhoms,

I would like to filter for indicators with the category "allow" or "optimize" only. How would you define the filter for that? I cannot find that much information regarding filtering using a processor. I hope my steps are correct? 

1. create a new prototype of the IPv4Generic processor
2. create infilters for that
   

   infilters:
   -   actions:
       - accept
       conditions:
       - __method == 'withdraw'
       name: accept withdraws
   -   actions:
       - accept
       conditions:
       - o365_category == 'Allow'
       name: accept o365_categoryAllow
   -   actions:
       - accept
       conditions:
       - o365_category == 'Optimize'
       name: accept o365_categoryOptimize
   -   actions:
       - drop
       name: drop all

3. create a processor node using the previously selfmade prototype
4. set as input the o365 miner
5. create a output / feed node using the HCGreenWithValue prototype & set as input the selfmade processor

Thanks a lot for your help!

[Potato-soup]

I have created a feed of O365 URLs filtering on expressRoute = True

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - o365_expressRoute == true
    name: accept o365_expressRoute
-   actions:
    - drop
    name: drop all

 All works fine...  However,

Comparing the results from Minemeld with the output from a bit of PowerShell, which does the same job. Minemeld does not include the URL *.outlook.office.com.  

(Invoke-WebRequest -Uri $EndpointUri  | ConvertFrom-Json) | % { 
    if ($_.expressRoute) { $_.urls }
     }

I can not see why.