Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.
As customers migrate to Office 365, they find themselves whitelisting a range of App-IDs for the various workloads they might use the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of any, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment
First, visit the MineMeld Resource Page and select the article (from the top right) about installing and running MineMeld appropriate to your environment. NOTE: if using the VMWare desktop instructions (read Running MineMeld On VMWare Desktop), you can go ahead with the "Super fast setup," but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of MineMeld.
Make note of MineMeld's IP address (from an ifconfig) as you’ll need it for the web interface in the next step.
MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a set of configurations you can import. Unzip the attached file MMO365-API_ConfigFiles.zip to get the following collection of configurations.
In this document we'll use the configuration named "o365-api-any-any.txt" that will set a graph to mine all ServiceAreas in all O365 Instances. But you might choose to use the any configuration that better suits your needs.
Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.
Take into account that this procedure will replace any configuration you might have with this new collection of nodes. Your old configuration will be lost.
Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.
After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.
Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:
Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated endpoints (as described in Managing Office 365 endpoints), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).
Click each of the output notes and make a note of the Feed Base URL.
Follow the instructions in the article Connecting PAN-OS to MineMeld using External Dynamic Lists to connect your PAN-OS Device with the lists provided by MineMeld
This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later. Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions,” then block to block all categories at once). Scroll to the bottom and allow only the external dynamic list of O365 URLs.
Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.
The following is the set of screenshots that ilustrate how the security policy should be created.
App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)
You may find (from using a catch-all rule with logging) that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination address of any. In the Service/URL category tab, insert the custom URL category from Step 5. The FQDNs will be present in that URL category and thus match this second rule
There are 2 ways of doing this.
Use a local Miner. This works on any Miner version:
Select only required endpoints. This requires MineMeld version 0.9.62+. An easy way to automatically remove all the 3rd party URLs is selecting only the O365 URLs marked as required by MSFT. MineMeld translates this attribute in the internal o365_required_list attribute (a list) and we can create a filter based on that:
infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- contains(o365_required_list, 'true') == true
name: accept o365 required indicators
- actions:
- drop
name: drop all
store_value: true
-
Does anyone know how to filter the results returned by the miners based on ID or Category? ...the fields highlighted in the screenshot below?
Any suggestion will be highly appreciated.
Hi,
I successfully imported the txt file to minemeld but I am getting this error
"__init__() got an unexpected keyword argument 'server_hostname'"
the feeds and indicators are not able to pull any IP addresses.
Please let me know what I can do to resolve the issue
