Enable Access to Office 365 with MineMeld [Updated]

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Audit
Last Reviewed: 09-01-2023 06:09 AM
Audited By: kiwi
L5 Sessionator
90% helpful (9/10)

Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.


 
This article describes a procedure that requires MineMeld version 0.9.50 or newer.

2018-09-25_17-21-13.png

 

Overview

As customers migrate to Office 365, they find themselves whitelisting a range of App-IDs for the various workloads they might use the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of any, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment

 

Step 1. Deploy MineMeld

First, visit the MineMeld Resource Page and select the article (from the top right) about installing and running MineMeld appropriate to your environment. NOTE: if using the VMWare desktop instructions (read Running MineMeld On VMWare Desktop), you can go ahead with the "Super fast setup," but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of MineMeld.

 

Make note of MineMeld's IP address (from an ifconfig) as you’ll need it for the web interface in the next step.

 

Step 2. Obtain & Import Configuration

MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a set of configurations you can import. Unzip the attached file MMO365-API_ConfigFiles.zip to get the following collection of configurations.

  • o365-api-any-any.txt
  • o365-api-skype-usgovdod.txt
  • o365-api-sharepoint-usgovdod.txt
  • o365-api-exchange-usgovdod.txt
  • o365-api-skype-usgovgcchigh.txt
  • o365-api-sharepoint-usgovgcchigh.txt
  • o365-api-exchange-usgovgcchigh.txt
  • o365-api-skype-china.txt
  • o365-api-sharepoint-china.txt
  • o365-api-exchange-china.txt
  • o365-api-skype-germany.txt
  • o365-api-sharepoint-germany.txt
  • o365-api-exchange-germany.txt
  • o365-api-skype-ww.txt
  • o365-api-sharepoint-ww.txt
  • o365-api-exchange-ww.txt
  • o365-api-any-usgovdod.txt
  • o365-api-any-usgovgcchigh.txt
  • o365-api-any-china.txt
  • o365-api-any-germany.txt
  • o365-api-any-ww.txt

In this document we'll use the configuration named "o365-api-any-any.txt" that will set a graph to mine all ServiceAreas in all O365 Instances. But you might choose to use the any configuration that better suits your needs.

 

Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.

 

MineMeld web interface.png

 

MineMeld config replace.png

Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost.

Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.

 

Step 3. Review Connection Graph and retrieve Feed Base URLs

After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.

 

MineMeld node view.png

 

Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:

 

MineMeld node show.png

 

 

Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated endpoints (as described in Managing Office 365 endpoints), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).

 

Click each of the output notes and make a note of the Feed Base URL.

 

MineMeld feed link.png

 

 

Step 4. Consume MineMeld’s output

Follow the instructions in the article Connecting PAN-OS to MineMeld using External Dynamic Lists to connect your PAN-OS Device with the lists provided by MineMeld

 

Step 5. Create a URL Filtering Profile

This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later. Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions,” then block to block all categories at once).  Scroll to the bottom and allow only the external dynamic list of O365 URLs.

 

URL Filtering Profile.png

 

Step 6: Create Security Policies

Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.

 

The following is the set of screenshots that ilustrate how the security policy should be created.

 

sr_01.png

 

sr_02.png

 

sr_03.png

 

sr_04.png

 

sr_05.png

 

sr_06.png

 

sr_07.png

FAQ

What applications should I list in the policy?

App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)

  • activesync
  • mapi-over-http
  • ms-exchange
  • ms-office365
  • ms-onedrive
  • rpc-over-http
  • soap
  • ssl
  • stun
  • web-browsing
  • webdav
  • ms-office365
  • office-live
  • office-on-demand
  • outlook-web-online
  • ms-lync-online
  • ms-lync-online-apps-sharing
  • sharepoint-online
  • ms-lync-online-file-transfer

 

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5. The FQDNs will be present in that URL category and thus match this second rule

 

How do I filter 3rd party URLs from the endpoint list?

There are 2 ways of doing this.

Use a local Miner. This works on any Miner version:

  • create a new Miner based on stdlib.localDB to be used a whitelist. The name of the Miner should start with "wl", example: wlSneaky3rdPartyURLs
  • connect the Miner to the URL aggregator for the O365 URL and commit
  • on the Web UI, under Nodes click on the new Miner and click on the table icon on the left

image (10).png

  • add the undesired URL in the indicator list (www.youtube.com in our case) and disable expiration. Click OKimage (11).png
  • The undesired is now removed from the URL list

Select only required endpoints. This requires MineMeld version 0.9.62+. An easy way to automatically remove all the 3rd party URLs is selecting only the O365 URLs marked as required by MSFT. MineMeld translates this attribute in the internal o365_required_list attribute (a list) and we can create a filter based on that:

  • go to the prototypes, search feedHCWithValue and click NEW (not CLONE)
  • give to the new prototype a meaningful name
  • paste this config:

 

 

 

 

 

 

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - contains(o365_required_list, 'true') == true
    name: accept o365 required indicators
-   actions:
    - drop
    name: drop all
store_value: true

 

 

 

 

 

 

  • use the new prototype to build a new output node and connect it to the URL aggregator
  • only required URLs will be placed in the new output node

-  

Rate this article:
(1)
Comments
L0 Member

Hi, thanks for this post.

I think a quick update needs to be done on Step 5

If i am setting "block" in the URL filtering for every other categories than EDL, I got access denied for a lot of things. F

or example, teams's URL teams.microsoft.com is matching the category "computer-and-internet-info", same for skype with "pip.skype.com" which is in category "internet-communication-and-telephony" etc. 

L1 Bithead

Hi

 

Thanks for the update. Quick question, we're a public body based in the UK; do we need to use the o365-api-any-any.txt configuration? Microsoft have a datacenter based here now

 

Cheers

 

Ben

L5 Sessionator

Hi @BriceCRUNCHANT what you are experiencing is due the fact PANOS, in case of a given URL matching multiple categories,  choses the one with the most severe action - https://live.paloaltonetworks.com/t5/Management-Articles/URL-Filtering-Order/ta-p/59334

 

It is safe to enable other categories like "computer-and-internet-info" and "internet-communication-and-telephony" (Step 5) because URL's in these categories will only be allowed if the content is provided from the O365's IPv4/IPv6 address space.

L5 Sessionator

Hi @benslade, the O365 endpoints (AKA the new REST API for MS O365 endpoints) is documented in https://support.office.com/en-gb/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728e...

 

If you follow the document you'll find the URL to check for available O365 instances. At the moment of writing this comment, the list does not contain a specific instance for UK Public. That means that "o365-api-any-any.txt" will definitely cover your case but that you can also try "o365-api-any-ww.txt" if you need to harden your policy.

 

[
  {
    "instance": "Worldwide",
    "latest": "2018080200"
  },
  {
    "instance": "USGovDoD",
    "latest": "2018073100"
  },
  {
    "instance": "USGovGCCHigh",
    "latest": "2018063000"
  },
  {
    "instance": "China",
    "latest": "2018073000"
  },
  {
    "instance": "Germany",
    "latest": "2018063000"
  }
]
L0 Member

Ciao to all, this is Marco. I'm asking an help about a paloalto 850 and minemeld.

I have 2 different internet connections attached to the firewall.

One is dedicate to servers traffic and office365. (and this is the default for the firewall)

One is dedicate to web browsing. (this is configured using a PBF).

After implementing minemeld, 365 traffic go away to te default connection (and is right).

The problem is on SKYPE4BUSINESS. Only chat is working. Audio and video calls, desktop sharing and sending request of contacts (to both skype and skype4business) are not working.

Please have you some suggestions?

Thanks indeed.

 

Ciao ciao,

 

Marco

 

L5 Sessionator

Hi @ConfindustriaBG,

 

have you been able to discover the reason that is blocking your users from using audio and video in Skype call? I guess it is because you're not enabling all needed applications (stun ...) but there may be many other explanations.

L2 Linker

Hi,

 

If you've got problems importing these configuration files, make sure you're running MineMeld 0.9.50 as it'll not work with 0.9.36 for example.

 

If you need to upgrade, run "sudo /usr/sbin/minemeld-auto-update" and it should be ok.

L1 Bithead

Just a note on the suggesting of importing of the Office 365 config and overwriting your existing config which is a bit bizarre!!!

 

When this article says "Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost." it literally means ANY config... no matter if its an existing security feed config etc, it will be ovewritten!!!

 

HOWEVER, fear not...

 

1. You should have taken a backup of the system before-hand right? E.g:

* A VM snapshot if running on a VM. 

* An export of the existing config to a text file.

 

2. Even if you do choose to OVERWRITE your config, you can roll it back by immediately pressing REVERT button in the Config section.

 

3. Despite what the article says, you do not need to OVERWRITE, but you can APPEND the config instead if you wish, therefore keeping your existing configs and complimenting them with the Office 365 config. - Just make sure you miners, processors and outputs aren't clashing. 

 

Remember - you can REVERT.

 

Once you're happy, then you can COMMIT.

L4 Transporter

RLJFRY - You sir, should get  a promotion for the above comment. Thank you

 

How are people proceeding with existing versions of minemeld that are not running .50 release?  Is it suggested to stick with your initial deployments, or have you gone to updating your minemeld instance and utilized the directions cited here?

L1 Bithead

@Sec101 Haha! Glad to help.

 

That's the beauty of community support 🙂 

 

 

L4 Transporter

We are seeing one of our sync servers trying to hit amazon ip's, and it's not matching policy.  It's hitting the CDN part that is mentioned below.  How are people handling this knowing that your URL list is allowing sites like dropbox/itunes/ and other 3rd party items?  Are people literrally only placing the URL profile and a desination of any for this?

 

 

What if there's still some O365 activity that is NOT hitting my new security policy?

You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5.  The FQDNs will be present in that URL category and thus match this second rule

L3 Networker

Hi guys, looks like there is an issue with the URL processor. Please could you check this out as it breaks OneNote.

 

The original input from Microsoft includes for instance two URLs "cdn.onenote.net" and "site-cdn.onenote.net" under "id": 271, and these two are aggregated to "*cdn.onenote.net" which is invalid as the wildcard and charaters cannot be in the same token.

 

In total there are 11 entires like this and the FireWall log shows that it skipped exactly 11

    "Office365-URL, 1, 1 url) Valid entries(578) lines skipped(11)"

    

*broadcast.officeapps.live.com
*cdn.onenote.net
*excel.officeapps.live.com
*onenote.officeapps.live.com
*powerpoint.officeapps.live.com
*view.officeapps.live.com
*visio.officeapps.live.com
*word-edit.officeapps.live.com
*word-view.officeapps.live.com
*-files.sharepoint.com
*-myfiles.sharepoint.com

  

L1 Bithead

on 7.121 I'm getting consistant URL access error...It is https, and I can ping the host from the CLI, not sure why I'm getting this.

L2 Linker

@LarsAtConsigas 

 

I know this is an old post, replying to benefit other users.

We ran into the same issue, talked to Support and this is expected behavior.

If you add "?v=panosurl" to the end of the Minemeld URL then you will get a list better formated to use on the firewall.

For example it will convert "*cdn.onenote.net" to two entries; "onenote.net" and "*.onenote.net".

L1 Bithead

Hello,

 

Where can I get the files? MMO365-API_ConfigFiles.zip? Thank you in advance!

L1 Bithead

Do you need help configuring Minemeld or just the MS API URLs to get to the IPv4/v6 and URL lists?

 

L1 Bithead

I just needed the list, Thank you. I just followed the directions I found online and got it up and running at this point.

 

I do have another question. Can I have multiple configs? When I imported the config from the .zip, it says I overwrote what was in there. Does that mean you can only have one config. Meaning only the o365 config. I was hoping to incorporate many others, i.e. Apple services/ip addresses. That is my second question. Is there a safe enable for Apple?

L2 Linker

  Does anyone know how to filter the results returned by the miners based on ID or Category? ...the fields highlighted in the screenshot below?

  Any suggestion will be highly appreciated.

 

Capture.PNG

 

 

 

 

 

L1 Bithead

Hi, 

The worldwide url and any-any list contains sites like youtube.com. How can i remove them?

 

I've read you have to disable "INTEGRATIONS" in the miner, but this didn't work. It did remove some entries, but youtube is still present

 

Microsoft states all url's need to be excluded from ssl decryption so using this list wil leave those 3rd parties also encrypted.

L1 Bithead

Is there a safely enable access to apple?

L1 Bithead

Last time I looked the "disable integrations" code looked for the string "integration" in the "notes" attribute. To filter out youtube and others the regex /integration|(3|thi)rd[- ]part(y|ies)/i has to be applied to this attribute.

 

L1 Bithead

Microsoft states that categories "Required" and "Allow" are incompatible with SSL decryption. All the integration stuff has:

    "category": "Default",
    "required": false,

So you could decrypt them. Altenatively you can specify URLs AND IP addresses in the "No Decryption" rule because MS does not list/know the IP addresses of those 3rd party services. Of course this would also affect MS services hosted on Akamai and other non-MS CDNs.

L1 Bithead

And how would one configure this regex ? Maybe you can attach an example config?

thanks in advance

 

L1 Bithead

This would be in the python code of the miner.

It would be great if we could implement this ourselves in nodes input or output conditions, but I don't know if this is possible. Can someone point to more detailed documentation, please?

 

L1 Bithead

Found this solution to remove youtube:

 

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-excluding-entries-from-URL-list/m...

 

But the palo alto implementation is flawed.

L7 Applicator

@wiresharky @ttsws 

The best way to filter out youtube and other 3rd parties, and select only "required" endpoints is building an output node with the right filters applied to the endpoints. I will add this to the article, something similar to:

infilters:
-   actions:
    - accept
    conditions:
    - __method == 'withdraw'
    name: accept withdraws
-   actions:
    - accept
    conditions:
    - contains(o365_required_list, 'true') == true
    name: accept o365 required indicators
-   actions:
    - drop
    name: drop all
store_value: true
L0 Member

Hello,

 

I face the same issue than this topic: https://live.paloaltonetworks.com/t5/MineMeld-Discussions/AutoFocus-MineMeld-with-Office-365/td-p/26...

 

A customer asks for decrypting OneDrive (Sharepoint) traffic (despite if it is not recommended by MS). Thus I've created new nodes depending on the 'o365-api.worldwide-sharepoint' prototype but I see in my outputs the URLs/IPv4/IPv6 belonging to the "Sharepoint" serviceArea + from the "Common" serviceArea.

 

Do you know a way to exclude the "Common" area from the miner?

 

Thanks

L1 Bithead

Hello, I have my minemeld running now and wanted to add o365. I followed the description here and it looks basically OK, but my o365 miner all have a red exclamation mark. The error says endpoints.office.com read timeout. In my firewall log I don´t see any blocked access to this site and the spamhaus default miner is also working.

 

What am I doing wrong?

 

---edit---

I think you can forget my post - I just found the URLs on a Microsoft website and they actually time out at the moment...

---edit---

 

Brgds Andreas

L0 Member

I echo comments by Deas.h

 

I have minemeld installed and have imported the o365-api-any-any.txt Office365 script but now have 5 miners that have a red exclamation mark against them.

States read timed out going to endpoints.microsoft.com, however I can manually browse to endpoints.microsoft.com

The firewall is not blocking anything but I can not get the miners to work

 

What am I doing wrong?

 

Cheers

 

Wozza

L7 Applicator

@Warren_Norman we are experiencing the same odd issue, it doesn't seem to be related to MineMeld as most of the MineMeld instances works as expected.

Could you try running this from MineMeld shell:

curl "https://endpoints.office.com/version?clientrequestid=3fb1f126-e221-490f-857e-c4332d38979a"

 

L1 Bithead

I also think there is no problem with minemeld. It is related to Microsoft. But who/where can we report this issue?

 

In the meantime the "o365-worldwide-any-miner" miner was able to download once 13 hours ago. And now only read timeout again.

 

And I found this on GitHub: https://github.com/PaloAltoNetworks/minemeld/issues/49 So this is a Microsoft issue. Why now and why for such a long time???

 

The curl command is doing now nothing for about 10 minutes...

 

Brgds Andreas

L7 Applicator

@Deas.h I have been in touch with Microsoft during the weekend, it seems that this is happening only on some specific locations/IPs around the world and it was difficult to reproduce.

@Deas.h@Warren_Norman  Could you send me an email to lmori at paloaltonetworks.com if you don't mind sharing your details with Microsoft?

 

Thanks!

Luigi

L0 Member

@lmori I run that and got 'TCP connection reset by peer', strange thing is I pasted that url into my browser and was able to download the json file?

L1 Bithead

For me it does not matter if I do it with curl or a browser. Both ways are not working from our company. But when I try it from my home internet connection it is working instantly. Strange...

 

Brgds Andreas

L1 Bithead

This will be painfully obvious once someone points it out. Where is this file attached and how do I get it?

 

"Unzip the attached file MMO365-API_ConfigFiles.zip to get the following collection of configurations." 

L3 Networker

@ccarter I had a hard time as well, its right before the comments section! I even searched for it and could not find it.

L0 Member

Minemeld should also use categorize parameters from new Endpoints and better usage with split tunneling configurations.

 

https://github.com/PaloAltoNetworks/minemeld-node-prototypes/issues/120

L7 Applicator

@daniel.sneto MineMeld is already exposing the parameters for each Endpoint and those can be used to create split tunnelling configs.

L4 Transporter

@lmori

 

Is that a manual process?  Is there an EDL that one can use for something like this?

L0 Member

Hi @lmori thanks for you help.

Can you explain better how can we use this categories also on Minemeld?

L2 Linker

Just a FYI about these o365 Feeds - they include URL's from other vendors, like Google, Evernote, Dropbox and some others.  We exempt the o365 from TLS decryption, and I found that our enforcement of Non-Company domains for G-suite were being allowed (i.e. personal Gmail)  Previously, personal G-mail had been denied using HTTP Header insertion on the PA, but looking into things further, these MS o365 URL list is brining in the following URL's from Google, which was causing no-decrypt on these, bypassing our policy for non company domain Gmail.  I had to add these to the dynamic URL exception list to get things working again.  In our case, it was the following URL's being brought in:

accounts.google.com
mail.google.com
play.google.com
www.google-analytics.com www.googleapis.com 

 Not sure what MS is trying to do here.  They must be trying to do some integration with Google from O365?  

L3 Networker

Can we get office-365 ip-address list from minemeld ?

We want to source-nat office-365 traffic on particular public ip.

L0 Member

Hi, 

 

I successfully imported the txt file to minemeld but I am getting this error 

"__init__() got an unexpected keyword argument 'server_hostname'"

 

the feeds and indicators are not able to pull any IP addresses. 

 

Please let me know what I can do to resolve the issue minemeld.PNG

 

 

L1 Bithead

can someone please provide step by step configuration of Mine mild  into Palo alto Firewall for dynamic updates of office 365.

Thanks

  • 217889 Views
  • 44 comments
  • 11 Likes
Register or Sign-in
Contributors
Labels
Article Dashboard
Version history
Last Updated:
‎12-14-2021 06:01 AM
Updated by: