How to Safely Enable access to Office 365 using MineMeld [Updated]

by xhoms ‎08-01-2018 07:44 AM - edited ‎09-26-2018 02:26 AM (15,298 Views)
This article describes a procedure that requires MineMeld version 0.9.50 or newer.

2018-09-25_17-21-13.png

 

Overview

As customers migrate to Office 365 they find themselves whitelisting a range of App-IDs for the various workloads they might use in the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of ‘any’, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment

 

Step 1. Deploy MineMeld

First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.

 

Make note of MineMeld's IP address (from an ifconfig) as you’ll need it for the Web UI in the next step.

 

Step 2. Obtain & Import Configuration

MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a set of configurations you can import. Just browse to https://paloaltonetworks.box.com/s/ywkh7rc2rj0kyl0qetr6m6ag3akxvvx6 to get the following collection of configurations.

  • o365-api-any-any.txt
  • o365-api-skype-usgovdod.txt
  • o365-api-sharepoint-usgovdod.txt
  • o365-api-exchange-usgovdod.txt
  • o365-api-skype-usgovgcchigh.txt
  • o365-api-sharepoint-usgovgcchigh.txt
  • o365-api-exchange-usgovgcchigh.txt
  • o365-api-skype-china.txt
  • o365-api-sharepoint-china.txt
  • o365-api-exchange-china.txt
  • o365-api-skype-germany.txt
  • o365-api-sharepoint-germany.txt
  • o365-api-exchange-germany.txt
  • o365-api-skype-ww.txt
  • o365-api-sharepoint-ww.txt
  • o365-api-exchange-ww.txt
  • o365-api-any-usgovdod.txt
  • o365-api-any-usgovgcchigh.txt
  • o365-api-any-china.txt
  • o365-api-any-germany.txt
  • o365-api-any-ww.txt

In this document we'll use the configuration named "o365-api-any-any.txt" that will set a graph to mine all ServiceAreas in all O365 Instances. But you might choose to use the any configuration that better suits your needs.

 

Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.

 

01_config_click.png

 

02_config_replace.png

 

 

Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost.

Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.

 

Step 3. Review Connection Graph and retrieve Feed Base URLs

After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.

 

03_node_click.png

 

 

Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:

 

04_node_show.png

 

 

Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated endpoints (as described in https://support.office.com/en-us/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728e...), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).

 

Click each of the Output notes and make a note of the Feed Base URL.

 

05_feed_link.png

 

 

Step 4. Consume MineMeld’s output

Follow the instructions in the article Connecting PAN-OS to MineMeld using External Dynamic Lists to connect your PANOS Device with the lists provided by MineMeld

 

Step 5. Create a URL Filtering Profile

This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later.  Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions”, then block to block all categories at once).  Scroll to the bottom and allow only the external dynamic list of O365 URLs.

 

05_url_filtering.png

 

Step 6: Create Security Policies

Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.

 

The following is the set of screenshots that ilustrate how the security policy should be created.

 

sr_01.png

 

sr_02.png

 

sr_03.png

 

sr_04.png

 

sr_05.png

 

sr_06.png

 

sr_07.png

FAQ

What applications should I list in the policy?

App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)

  • activesync
  • mapi-over-http
  • ms-exchange
  • ms-office365
  • ms-onedrive
  • rpc-over-http
  • soap
  • ssl
  • stun
  • web-browsing
  • webdav
  • ms-office365
  • office-live
  • office-on-demand
  • outlook-web-online
  • ms-lync-online
  • ms-lync-online-apps-sharing
  • sharepoint-online
  • ms-lync-online-file-transfer

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5.  The FQDNs will be present in that URL category and thus match this second rule

Comments
by BriceCRUNCHANT
on ‎08-14-2018 05:50 AM

Hi, thanks for this post.

I think a quick update needs to be done on Step 5

If i am setting "block" in the URL filtering for every other categories than EDL, I got access denied for a lot of things. F

or example, teams's URL teams.microsoft.com is matching the category "computer-and-internet-info", same for skype with "pip.skype.com" which is in category "internet-communication-and-telephony" etc. 

by benslade
on ‎08-17-2018 04:44 AM

Hi

 

Thanks for the update. Quick question, we're a public body based in the UK; do we need to use the o365-api-any-any.txt configuration? Microsoft have a datacenter based here now

 

Cheers

 

Ben

by xhoms
‎08-21-2018 12:09 AM - edited ‎08-21-2018 12:10 AM

Hi @BriceCRUNCHANT what you are experiencing is due the fact PANOS, in case of a given URL matching multiple categories,  choses the one with the most severe action - https://live.paloaltonetworks.com/t5/Management-Articles/URL-Filtering-Order/ta-p/59334

 

It is safe to enable other categories like "computer-and-internet-info" and "internet-communication-and-telephony" (Step 5) because URL's in these categories will only be allowed if the content is provided from the O365's IPv4/IPv6 address space.

by xhoms
on ‎08-21-2018 12:19 AM

Hi @benslade, the O365 endpoints (AKA the new REST API for MS O365 endpoints) is documented in https://support.office.com/en-gb/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728e...

 

If you follow the document you'll find the URL to check for available O365 instances. At the moment of writing this comment, the list does not contain a specific instance for UK Public. That means that "o365-api-any-any.txt" will definitely cover your case but that you can also try "o365-api-any-ww.txt" if you need to harden your policy.

 

[
  {
    "instance": "Worldwide",
    "latest": "2018080200"
  },
  {
    "instance": "USGovDoD",
    "latest": "2018073100"
  },
  {
    "instance": "USGovGCCHigh",
    "latest": "2018063000"
  },
  {
    "instance": "China",
    "latest": "2018073000"
  },
  {
    "instance": "Germany",
    "latest": "2018063000"
  }
]
by ConfindustriaBG
on ‎09-13-2018 02:34 AM

Ciao to all, this is Marco. I'm asking an help about a paloalto 850 and minemeld.

I have 2 different internet connections attached to the firewall.

One is dedicate to servers traffic and office365. (and this is the default for the firewall)

One is dedicate to web browsing. (this is configured using a PBF).

After implementing minemeld, 365 traffic go away to te default connection (and is right).

The problem is on SKYPE4BUSINESS. Only chat is working. Audio and video calls, desktop sharing and sending request of contacts (to both skype and skype4business) are not working.

Please have you some suggestions?

Thanks indeed.

 

Ciao ciao,

 

Marco

 

by xhoms
on ‎09-14-2018 03:10 PM

Hi @ConfindustriaBG,

 

have you been able to discover the reason that is blocking your users from using audio and video in Skype call? I guess it is because you're not enabling all needed applications (stun ...) but there may be many other explanations.

by brrenaud
‎09-25-2018 01:48 AM - edited ‎09-25-2018 01:49 AM

Hi,

 

If you've got problems importing these configuration files, make sure you're running MineMeld 0.9.50 as it'll not work with 0.9.36 for example.

 

If you need to upgrade, run "sudo /usr/sbin/minemeld-auto-update" and it should be ok.

by RLJFRY
on ‎10-09-2018 09:37 AM

Just a note on the suggesting of importing of the Office 365 config and overwriting your existing config which is a bit bizarre!!!

 

When this article says "Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost." it literally means ANY config... no matter if its an existing security feed config etc, it will be ovewritten!!!

 

HOWEVER, fear not...

 

1. You should have taken a backup of the system before-hand right? E.g:

* A VM snapshot if running on a VM. 

* An export of the existing config to a text file.

 

2. Even if you do choose to OVERWRITE your config, you can roll it back by immediately pressing REVERT button in the Config section.

 

3. Despite what the article says, you do not need to OVERWRITE, but you can APPEND the config instead if you wish, therefore keeping your existing configs and complimenting them with the Office 365 config. - Just make sure you miners, processors and outputs aren't clashing. 

 

Remember - you can REVERT.

 

Once you're happy, then you can COMMIT.

by Sec101
a month ago - last edited a month ago

RLJFRY - You sir, should get  a promotion for the above comment. Thank you

 

How are people proceeding with existing versions of minemeld that are not running .50 release?  Is it suggested to stick with your initial deployments, or have you gone to updating your minemeld instance and utilized the directions cited here?

by RLJFRY
a month ago

@Sec101 Haha! Glad to help.

 

That's the beauty of community support :) 

 

 

by Sec101
2 weeks ago

We are seeing one of our sync servers trying to hit amazon ip's, and it's not matching policy.  It's hitting the CDN part that is mentioned below.  How are people handling this knowing that your URL list is allowing sites like dropbox/itunes/ and other 3rd party items?  Are people literrally only placing the URL profile and a desination of any for this?

 

 

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5.  The FQDNs will be present in that URL category and thus match this second rule

by LarsAtConsigas
Saturday

Hi guys, looks like there is an issue with the URL processor. Please could you check this out as it breaks OneNote.

 

The original input from Microsoft includes for instance two URLs "cdn.onenote.net" and "site-cdn.onenote.net" under "id": 271, and these two are aggregated to "*cdn.onenote.net" which is invalid as the wildcard and charaters cannot be in the same token.

 

In total there are 11 entires like this and the FireWall log shows that it skipped exactly 11

    "Office365-URL, 1, 1 url) Valid entries(578) lines skipped(11)"

    

*broadcast.officeapps.live.com
*cdn.onenote.net
*excel.officeapps.live.com
*onenote.officeapps.live.com
*powerpoint.officeapps.live.com
*view.officeapps.live.com
*visio.officeapps.live.com
*word-edit.officeapps.live.com
*word-view.officeapps.live.com
*-files.sharepoint.com
*-myfiles.sharepoint.com

  

Ask Questions Get Answers Join the Live Community
Labels
Contributors