Enable Access to Office 365 with MineMeld [Updated]

Printer Friendly Page
This article describes a procedure that requires MineMeld version 0.9.50 or newer.

2018-09-25_17-21-13.png

 

Overview

As customers migrate to Office 365, they find themselves whitelisting a range of App-IDs for the various workloads they might use the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of any, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment

 

Step 1. Deploy MineMeld

First, visit the MineMeld Resource Page and select the article (from the top right) about installing and running MineMeld appropriate to your environment. NOTE: if using the VMWare desktop instructions (read Running MineMeld On VMWare Desktop), you can go ahead with the "Super fast setup," but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of MineMeld.

 

Make note of MineMeld's IP address (from an ifconfig) as you’ll need it for the web interface in the next step.

 

Step 2. Obtain & Import Configuration

MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a set of configurations you can import. Unzip the attached file MMO365-API_ConfigFiles.zip to get the following collection of configurations.

  • o365-api-any-any.txt
  • o365-api-skype-usgovdod.txt
  • o365-api-sharepoint-usgovdod.txt
  • o365-api-exchange-usgovdod.txt
  • o365-api-skype-usgovgcchigh.txt
  • o365-api-sharepoint-usgovgcchigh.txt
  • o365-api-exchange-usgovgcchigh.txt
  • o365-api-skype-china.txt
  • o365-api-sharepoint-china.txt
  • o365-api-exchange-china.txt
  • o365-api-skype-germany.txt
  • o365-api-sharepoint-germany.txt
  • o365-api-exchange-germany.txt
  • o365-api-skype-ww.txt
  • o365-api-sharepoint-ww.txt
  • o365-api-exchange-ww.txt
  • o365-api-any-usgovdod.txt
  • o365-api-any-usgovgcchigh.txt
  • o365-api-any-china.txt
  • o365-api-any-germany.txt
  • o365-api-any-ww.txt

In this document we'll use the configuration named "o365-api-any-any.txt" that will set a graph to mine all ServiceAreas in all O365 Instances. But you might choose to use the any configuration that better suits your needs.

 

Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.

 

MineMeld web interface.png

 

MineMeld config replace.png

Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost.

Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.

 

Step 3. Review Connection Graph and retrieve Feed Base URLs

After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.

 

MineMeld node view.png

 

Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:

 

MineMeld node show.png

 

 

Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated endpoints (as described in Managing Office 365 endpoints), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).

 

Click each of the output notes and make a note of the Feed Base URL.

 

MineMeld feed link.png

 

 

Step 4. Consume MineMeld’s output

Follow the instructions in the article Connecting PAN-OS to MineMeld using External Dynamic Lists to connect your PAN-OS Device with the lists provided by MineMeld

 

Step 5. Create a URL Filtering Profile

This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later. Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions,” then block to block all categories at once).  Scroll to the bottom and allow only the external dynamic list of O365 URLs.

 

URL Filtering Profile.png

 

Step 6: Create Security Policies

Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.

 

The following is the set of screenshots that ilustrate how the security policy should be created.

 

sr_01.png

 

sr_02.png

 

sr_03.png

 

sr_04.png

 

sr_05.png

 

sr_06.png

 

sr_07.png

FAQ

What applications should I list in the policy?

App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)

  • activesync
  • mapi-over-http
  • ms-exchange
  • ms-office365
  • ms-onedrive
  • rpc-over-http
  • soap
  • ssl
  • stun
  • web-browsing
  • webdav
  • ms-office365
  • office-live
  • office-on-demand
  • outlook-web-online
  • ms-lync-online
  • ms-lync-online-apps-sharing
  • sharepoint-online
  • ms-lync-online-file-transfer

 

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5. The FQDNs will be present in that URL category and thus match this second rule

 

How do I filter 3rd party URLs from the endpoint list?

There are 2 ways of doing this.

Use a local Miner. This works on any Miner version:

  • create a new Miner based on stdlib.localDB to be used a whitelist. The name of the Miner should start with "wl", example: wlSneaky3rdPartyURLs
  • connect the Miner to the URL aggregator for the O365 URL and commit
  • on the Web UI, under Nodes click on the new Miner and click on the table icon on the left

image (10).png

  • add the undesired URL in the indicator list (www.youtube.com in our case) and disable expiration. Click OKimage (11).png
  • The undesired is now removed from the URL list

Select only required endpoints. This requires MineMeld version 0.9.62+. An easy way to automatically remove all the 3rd party URLs is selecting only the O365 URLs marked as required by MSFT. MineMeld translates this attribute in the internal o365_required_list attribute (a list) and we can create a filter based on that:

  • go to the prototypes, search feedHCWithValue and click NEW (not CLONE)
  • give to the new prototype a meaningful name
  • paste this config:
infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - contains(o365_required_list, 'true') == true
    name: accept o365 required indicators
-   actions:
    - drop
    name: drop all
store_value: true
  • use the new prototype to build a new output node and connect it to the URL aggregator
  • only required URLs will be placed in the new output node

-  

Comments

Hi, thanks for this post.

I think a quick update needs to be done on Step 5

If i am setting "block" in the URL filtering for every other categories than EDL, I got access denied for a lot of things. F

or example, teams's URL teams.microsoft.com is matching the category "computer-and-internet-info", same for skype with "pip.skype.com" which is in category "internet-communication-and-telephony" etc. 

Hi

 

Thanks for the update. Quick question, we're a public body based in the UK; do we need to use the o365-api-any-any.txt configuration? Microsoft have a datacenter based here now

 

Cheers

 

Ben

Hi @BriceCRUNCHANT what you are experiencing is due the fact PANOS, in case of a given URL matching multiple categories,  choses the one with the most severe action - https://live.paloaltonetworks.com/t5/Management-Articles/URL-Filtering-Order/ta-p/59334

 

It is safe to enable other categories like "computer-and-internet-info" and "internet-communication-and-telephony" (Step 5) because URL's in these categories will only be allowed if the content is provided from the O365's IPv4/IPv6 address space.

Hi @benslade, the O365 endpoints (AKA the new REST API for MS O365 endpoints) is documented in https://support.office.com/en-gb/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728e...

 

If you follow the document you'll find the URL to check for available O365 instances. At the moment of writing this comment, the list does not contain a specific instance for UK Public. That means that "o365-api-any-any.txt" will definitely cover your case but that you can also try "o365-api-any-ww.txt" if you need to harden your policy.

 

[
  {
    "instance": "Worldwide",
    "latest": "2018080200"
  },
  {
    "instance": "USGovDoD",
    "latest": "2018073100"
  },
  {
    "instance": "USGovGCCHigh",
    "latest": "2018063000"
  },
  {
    "instance": "China",
    "latest": "2018073000"
  },
  {
    "instance": "Germany",
    "latest": "2018063000"
  }
]

Ciao to all, this is Marco. I'm asking an help about a paloalto 850 and minemeld.

I have 2 different internet connections attached to the firewall.

One is dedicate to servers traffic and office365. (and this is the default for the firewall)

One is dedicate to web browsing. (this is configured using a PBF).

After implementing minemeld, 365 traffic go away to te default connection (and is right).

The problem is on SKYPE4BUSINESS. Only chat is working. Audio and video calls, desktop sharing and sending request of contacts (to both skype and skype4business) are not working.

Please have you some suggestions?

Thanks indeed.

 

Ciao ciao,

 

Marco

 

Hi @ConfindustriaBG,

 

have you been able to discover the reason that is blocking your users from using audio and video in Skype call? I guess it is because you're not enabling all needed applications (stun ...) but there may be many other explanations.

Hi,

 

If you've got problems importing these configuration files, make sure you're running MineMeld 0.9.50 as it'll not work with 0.9.36 for example.

 

If you need to upgrade, run "sudo /usr/sbin/minemeld-auto-update" and it should be ok.

Just a note on the suggesting of importing of the Office 365 config and overwriting your existing config which is a bit bizarre!!!

 

When this article says "Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost." it literally means ANY config... no matter if its an existing security feed config etc, it will be ovewritten!!!

 

HOWEVER, fear not...

 

1. You should have taken a backup of the system before-hand right? E.g:

* A VM snapshot if running on a VM. 

* An export of the existing config to a text file.

 

2. Even if you do choose to OVERWRITE your config, you can roll it back by immediately pressing REVERT button in the Config section.

 

3. Despite what the article says, you do not need to OVERWRITE, but you can APPEND the config instead if you wish, therefore keeping your existing configs and complimenting them with the Office 365 config. - Just make sure you miners, processors and outputs aren't clashing. 

 

Remember - you can REVERT.

 

Once you're happy, then you can COMMIT.

RLJFRY - You sir, should get  a promotion for the above comment. Thank you

 

How are people proceeding with existing versions of minemeld that are not running .50 release?  Is it suggested to stick with your initial deployments, or have you gone to updating your minemeld instance and utilized the directions cited here?

@Sec101 Haha! Glad to help.

 

That's the beauty of community support :) 

 

 

We are seeing one of our sync servers trying to hit amazon ip's, and it's not matching policy.  It's hitting the CDN part that is mentioned below.  How are people handling this knowing that your URL list is allowing sites like dropbox/itunes/ and other 3rd party items?  Are people literrally only placing the URL profile and a desination of any for this?

 

 

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5.  The FQDNs will be present in that URL category and thus match this second rule

Hi guys, looks like there is an issue with the URL processor. Please could you check this out as it breaks OneNote.

 

The original input from Microsoft includes for instance two URLs "cdn.onenote.net" and "site-cdn.onenote.net" under "id": 271, and these two are aggregated to "*cdn.onenote.net" which is invalid as the wildcard and charaters cannot be in the same token.

 

In total there are 11 entires like this and the FireWall log shows that it skipped exactly 11

    "Office365-URL, 1, 1 url) Valid entries(578) lines skipped(11)"

    

*broadcast.officeapps.live.com
*cdn.onenote.net
*excel.officeapps.live.com
*onenote.officeapps.live.com
*powerpoint.officeapps.live.com
*view.officeapps.live.com
*visio.officeapps.live.com
*word-edit.officeapps.live.com
*word-view.officeapps.live.com
*-files.sharepoint.com
*-myfiles.sharepoint.com

  

on 7.121 I'm getting consistant URL access error...It is https, and I can ping the host from the CLI, not sure why I'm getting this.

@LarsAtConsigas 

 

I know this is an old post, replying to benefit other users.

We ran into the same issue, talked to Support and this is expected behavior.

If you add "?v=panosurl" to the end of the Minemeld URL then you will get a list better formated to use on the firewall.

For example it will convert "*cdn.onenote.net" to two entries; "onenote.net" and "*.onenote.net".

Hello,

 

Where can I get the files? MMO365-API_ConfigFiles.zip? Thank you in advance!

Do you need help configuring Minemeld or just the MS API URLs to get to the IPv4/v6 and URL lists?

 

I just needed the list, Thank you. I just followed the directions I found online and got it up and running at this point.

 

I do have another question. Can I have multiple configs? When I imported the config from the .zip, it says I overwrote what was in there. Does that mean you can only have one config. Meaning only the o365 config. I was hoping to incorporate many others, i.e. Apple services/ip addresses. That is my second question. Is there a safe enable for Apple?

  Does anyone know how to filter the results returned by the miners based on ID or Category? ...the fields highlighted in the screenshot below?

  Any suggestion will be highly appreciated.

 

Capture.PNG

 

 

 

 

 

Hi, 

The worldwide url and any-any list contains sites like youtube.com. How can i remove them?

 

I've read you have to disable "INTEGRATIONS" in the miner, but this didn't work. It did remove some entries, but youtube is still present

 

Microsoft states all url's need to be excluded from ssl decryption so using this list wil leave those 3rd parties also encrypted.

Is there a safely enable access to apple?

Last time I looked the "disable integrations" code looked for the string "integration" in the "notes" attribute. To filter out youtube and others the regex /integration|(3|thi)rd[- ]part(y|ies)/i has to be applied to this attribute.

 

Microsoft states that categories "Required" and "Allow" are incompatible with SSL decryption. All the integration stuff has:

    "category": "Default",
    "required": false,

So you could decrypt them. Altenatively you can specify URLs AND IP addresses in the "No Decryption" rule because MS does not list/know the IP addresses of those 3rd party services. Of course this would also affect MS services hosted on Akamai and other non-MS CDNs.

And how would one configure this regex ? Maybe you can attach an example config?

thanks in advance

 

This would be in the python code of the miner.

It would be great if we could implement this ourselves in nodes input or output conditions, but I don't know if this is possible. Can someone point to more detailed documentation, please?

 

Found this solution to remove youtube:

 

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-excluding-entries-from-URL-list/m...

 

But the palo alto implementation is flawed.

@wiresharky @ttsws 

The best way to filter out youtube and other 3rd parties, and select only "required" endpoints is building an output node with the right filters applied to the endpoints. I will add this to the article, something similar to:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - contains(o365_required_list, 'true') == true
    name: accept o365 required indicators
-   actions:
    - drop
    name: drop all
store_value: true

Hello,

 

I face the same issue than this topic: https://live.paloaltonetworks.com/t5/MineMeld-Discussions/AutoFocus-MineMeld-with-Office-365/td-p/26...

 

A customer asks for decrypting OneDrive (Sharepoint) traffic (despite if it is not recommended by MS). Thus I've created new nodes depending on the 'o365-api.worldwide-sharepoint' prototype but I see in my outputs the URLs/IPv4/IPv6 belonging to the "Sharepoint" serviceArea + from the "Common" serviceArea.

 

Do you know a way to exclude the "Common" area from the miner?

 

Thanks

Ask Questions Get Answers Join the Live Community
Version history
Revision #:
15 of 15
Last update:
a week ago
Updated by:
 
Contributors