This document provides an example of how to configure a custom miner prototype in MineMeld in order to retrieve an external threat feed. The feed will need to be manipulated through regex expressions to only include the portions which are readable by the Palo Alto Networks firewall.
Fusion for MAC
Tested with version 10.1.3
Tested with V0.9.50
Palo Alto Networks Virtual Firewall
Tested with VM50 PANOS 8.1.2
Use Case Diagram
The first step is to create a custom miner prototype. This prototype defines the external feed location as well as any custom regex required to pull out what is necessary (and remove what is not necessary) for the firewall to read it as an external dynamic list (EDL).
After logging into your instance of MineMeld, click the config menu-bar option to see the current configured items.
In the lower right-hand portion of the page, select the icon to “browse prototypes”.
Search for a prototype miner whose type matches the type of list you wish to create (i.e. IP4, Domain, URL). In this example, I selected itcertpa.URLS in order to create a customer URL miner prototype.
Click on the desired prototype to see the details. Then select “New” to create a new prototype based on this specific miner.
Modify the NAME and CONFIG areas as needed.
In this example, I want to bring in the ransomware tracker feed located at:
In addition, the regex will need to be modified in order to strip the http:// and https:// from the IOC’s so the firewall EDL can read the output. The ignore_regex field will be used to ignore any lines with the # symbol (the entire line).
Click OK to save the new prototype.
The next step is to create a new miner using the new prototype.
Go to the Config area.
Click on the “eye” icon in the lower right in order to change to expert mode. Once in expert mode, a plus icon will appear allowing you to add a MineMeld node.
Select the plus, provide the new node with a name. For the PROTOTYPE drop-down, select the new prototype previously created.
Select OK to save the new miner node.
Next, create a new Aggregator node (also known as a processor node). This node will aggregate one or more miner feeds, perform de-duplication, and prepare the data to be used by an output node.
In the Config section, select the icon to see all of the prototypes.
In the search field, type “processor” to see all of the processor prototypes. Look for one that matches the miner prototype created previously. In this example, I found stdlib.aggregatorURL. Once you find the aggregator example you wish to use, select it and then select “NEW” in the upper right-hand portion of the page to create a new aggregator node based on the one you found.
Give it a name and optionally, edit the CONFIG portion to remove any conditions that may not apply to your aggregator. In this example, I removed the area within the orange square. You may also add additional parameters depending on what you want your aggregated list to look like.
The next step is to create an aggregator node based on the new aggregator prototype just created. Go back to Config, enter expert mode, and select the plus to add a new aggregator node.
Give it a name, and for the PROTOTYPE drop-down, select the prototype just created. For the INPUTS field, select the custom miner node created in the first step.
The last node to be created is the Output node. This node will use the aggregated list and publish it to MineMeld’s internal web server so the firewall can read the final list and use it in a policy.
From the Config area, select the icon to see the prototypes. In the search field, look for “output”. Find one similar to what you want your output to look like. In this example, I used stdlib.feedGreenWithValue. Select the prototype and select “New” to create a new output based on the one selected.
Give it a name and optionally edit the CONFIG portion. In this example, I removed the portion within the orange square.
Go back to Config, enter expert mode, and select the plus to create a new output node based on the prototype just created. Give it a name, and select the output prototype in the dropdown. For the input, select the custom aggregator/processor node previously created.
Select OK to save.
You should see all three of the custom nodes created.
When ready, select COMMIT in the upper left-hand corner to save the nodes and put them to work.
To see if the list has been created, go to nodes.
Click the Output node you created and notice the FEED BASE URL link. Open the link to see the published list that the firewall will read. See a screenshot of the ransomware list below. Notice that the list no longer contains http:// or https:// references due to the regex working as expected.
The list is now ready to be consumed by the firewall.
The firewall configuration is much easier. Browse to your Palo Alto Networks firewall and go to Objects > External Dynamic Lists and select the Add button in the lower left-hand portion of the screen.
For Type, select the appropriate type for the node type created in MineMeld. Copy the FEED BASE URL from MineMeld and paste it into Source. Optionally, Test by clicking the Test Source URL button. Click OK to save.
The final step is to use the EDL within a policy. Go to Policies > Security and add a new rule (or modify an existing rule) where you want the policy to take effect.
In the Destination tab, under Destination Address, click Add and select the EDL just created.
Commit the config.
Using MineMeld is a powerful and easy way to bring in 3 rd party threat feeds based on IP, URL, and Domain. Using these feeds in your security policy is as easy as pointing the firewall to the published list and referring to the list in a policy. There are many use cases for EDL’s in both positive and negative enforcement scenarios. See the Live link below for additional ideas on incorporating EDL’s with MineMeld into your enterprise security operations.
To learn more about the free MineMeld tool:
To learn more about External Dynamic Lists:
Microsoft announced a new WEB Service that will deprecate the dynamic XML document used by the miners listed in this document. A new class and corresponding set of MineMeld prototypes was introduced in version 0.9.50 to deal with the new WEB Service.
To to safely enable access to Office 365 please follow the instructions in the updated document at https://live.paloaltonetworks.com/t5/MineMeld-Articles/How-to-Safely-Enable-access-to-Office-365-using-MineMeld-Updated/ta-p/224148
As customers migrate to Office 365 they find themselves whitelisting a range of App-IDs for the various workloads they might use in the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of ‘any’, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment.
Step 1. Deploy MineMeld
First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions ( https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038 ) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of Minemeld.
Make note of MineMelds IP address (from an ifconfig) as you’ll need it for the Web UI in the next step.
Step 2. Obtain & Import Configuration
MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a configuration you can import, simply download it from https://paloaltonetworks.app.box.com/s/4ubmkgrq72a8mdd24j733ddqdgbkyvv4 and open it in a text editor.
NOTE: for a minimal config collecting all the IPv4s, IPv6 and URLs of all the O365 products download this instead: https://paloaltonetworks.box.com/s/gndwe5rzheg1ekwplxb4m3mrpcf5k41f
Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.
This will bring up the IMPORT CONFIGURATION window. Copy and paste all the text from the .yml file you downloaded in step 2 into here and click Replace (or Append, if you have already configured this instance of Minemeld for another purpose.
Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.
Can't see an IMPORT button ? This is simply because you are using an older version of MineMeld. If you cannot upgrade for whatever reason, follow step 2a below instead. If not, carry on to step 3.
Step 2a. Importing configuration for an OLDER version of MineMeld only
nb: Skip this step if you were able to import using the web interface as above!
SCP the .yml file you downloaded in Step 2 to your MineMeld instance. For example, on a Mac, run the following with the default password rsplizardspock
$ scp ./office365-config.yml firstname.lastname@example.org:
To replace the configuration of a fresh install, SSH into your MineMeld instance (again as the ubuntu user) and run the following command:
$ sudo -u minemeld cp office365-config.yml /opt/minemeld/local/config/committed-config.yml
Or, to append an existing configuration (ie. you have other configuration you would like to keep such as the default Spamhaus polling), run the following command or manually append the contents of office365-config.yml to the end of committed-config.yml yourself in a text editor:
$ sudo -u minemeld cat office365-config.yml >> /opt/minemeld/local/config/committed-config.yml
Now run the command to restart MineMeld:
sudo service minemeld restart
Step 3. Review Connection Graph and retrieve Feed Base URLs
After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.
Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:
Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated XML File (direct link for your reference: https://support.content.office.net/en-us/static/O365IPAddresses.xml ), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).
Click each of the Output notes and make a note of the Feed Base URL.
Step 4. Consume MineMeld’s output
Log into your firewall (or Panorama) and go to Objects > External Dynamic Lists (or Objects > Dynamic Block Lists if using PAN-OS prior to v7.1). Click Add and create Dynamic IP address lists and URL lists to ‘subscribe’ to each of outputs created in the previous step. In my example below, I have created three dynamic lists matching the three Minemeld outputs above.
Step 5. Create a URL Filtering Profile
This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later. Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions”, then block to block all categories at once). Scroll to the bottom and allow only the external dynamic list of O365 URLs.
Step 6: Create Security Policies
Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.
App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)
What if there's still some O365 activity that is NOT hitting my new security policy?
You may find from using a catch-all rule with logging, that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. The following URL will allow you to confirm if this is the case and whether you need to widen your whitelist to allow for these CDNs.
To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination of any. In the Service/URL category tab, insert the custom URL category from Step 5. The FQDNs will be present in that URL category and thus match this second rule.