Using MineMeld to Create a Custom Miner

Printer Friendly Page

Introduction

This document provides an example of how to configure a custom miner prototype in MineMeld in order to retrieve an external threat feed. The feed will need to be manipulated through regex expressions to only include the portions which are readable by the Palo Alto Networks firewall.

 

 

Requirements

 

Required Item

Notes

Fusion for MAC

Tested with version 10.1.3

MineMeld VM

Tested with V0.9.50

Palo Alto Networks Virtual Firewall

Tested with VM50 PANOS 8.1.2

 

Use Case Diagram

 

Picture1.png

 

Configuration

MineMeld

 

The first step is to create a custom miner prototype.  This prototype defines the external feed location as well as any custom regex required to pull out what is necessary (and remove what is not necessary) for the firewall to read it as an external dynamic list (EDL).

 

After logging into your instance of MineMeld, click the config menu-bar option to see the current configured items.

 

Picture2.png

 

 

In the lower right-hand portion of the page, select the icon to “browse prototypes”.

 

Picture3.png

 

Search for a prototype miner whose type matches the type of list you wish to create (i.e. IP4, Domain, URL). In this example, I selected itcertpa.URLS in order to create a customer URL miner prototype.

 

Picture4.png

 

 

Click on the desired prototype to see the details. Then select “New” to create a new prototype based on this specific miner.

 

 

Picture5.png

 

Modify the NAME and CONFIG areas as needed.

 

 

Picture6.png

 

In this example, I want to bring in the ransomware tracker feed located at:

https://ransomwaretracker.abuse.ch/

 

In addition, the regex will need to be modified in order to strip the http:// and https:// from the IOC’s so the firewall EDL can read the output. The ignore_regex field will be used to ignore any lines with the # symbol (the entire line).

 

Picture7.png

 

Click OK to save the new prototype.

 

The next step is to create a new miner using the new prototype.

 

Go to the Config area.

 

Picture8.png

 

 

Click on the “eye” icon in the lower right in order to change to expert mode. Once in expert mode, a plus icon will appear allowing you to add a MineMeld node.

 

Picture9.png

 

Select the plus, provide the new node with a name. For the PROTOTYPE drop-down, select the new prototype previously created.

 

Picture10.png

 

Select OK to save the new miner node.

 

Next, create a new Aggregator node (also known as a processor node).  This node will aggregate one or more miner feeds, perform de-duplication, and prepare the data to be used by an output node.

 

In the Config section, select the icon to see all of the prototypes.

 

Picture11.png

 

In the search field, type “processor” to see all of the processor prototypes. Look for one that matches the miner prototype created previously. In this example, I found stdlib.aggregatorURL. Once you find the aggregator example you wish to use, select it and then select “NEW” in the upper right-hand portion of the page to create a new aggregator node based on the one you found.

 

Give it a name and optionally, edit the CONFIG portion to remove any conditions that may not apply to your aggregator. In this example, I removed the area within the orange square. You may also add additional parameters depending on what you want your aggregated list to look like.

 

Picture12.png

 

The next step is to create an aggregator node based on the new aggregator prototype just created. Go back to Config, enter expert mode, and select the plus to add a new aggregator node.

 

Give it a name, and for the PROTOTYPE drop-down, select the prototype just created. For the INPUTS field, select the custom miner node created in the first step.

 

Picture13.png

 

The last node to be created is the Output node. This node will use the aggregated list and publish it to MineMeld’s internal web server so the firewall can read the final list and use it in a policy.

 

From the Config area, select the icon to see the prototypes. In the search field, look for “output”. Find one similar to what you want your output to look like. In this example, I used stdlib.feedGreenWithValue. Select the prototype and select “New” to create a new output based on the one selected.

 

Give it a name and optionally edit the CONFIG portion. In this example, I removed the portion within the orange square.

 

 

Picture14.png

 

Go back to Config, enter expert mode, and select the plus to create a new output node based on the prototype just created. Give it a name, and select the output prototype in the dropdown. For the input, select the custom aggregator/processor node previously created.

 

Picture15.png

 

Select OK to save.

 

You should see all three of the custom nodes created.

 

Picture16.png

 

When ready, select COMMIT in the upper left-hand corner to save the nodes and put them to work.

 

To see if the list has been created, go to nodes.

 

Picture17.png

 

Click the Output node you created and notice the FEED BASE URL link. Open the link to see the published list that the firewall will read. See a screenshot of the ransomware list below. Notice that the list no longer contains http:// or https:// references due to the regex working as expected.

 

Picture18.png

 

Picture19.pngPicture20.png

 

The list is now ready to be consumed by the firewall.

 

Firewall

 

The firewall configuration is much easier. Browse to your Palo Alto Networks firewall and go to Objects > External Dynamic Lists and select the Add button in the lower left-hand portion of the screen.

 

Picture21.png

 

For Type, select the appropriate type for the node type created in MineMeld. Copy the FEED BASE URL from MineMeld and paste it into Source. Optionally, Test by clicking the Test Source URL button. Click OK to save.

 

Picture22.png

 

The final step is to use the EDL within a policy. Go to Policies > Security and add a new rule (or modify an existing rule) where you want the policy to take effect.

Picture23.png

 

In the Destination tab, under Destination Address, click Add and select the EDL just created.

 

Picture24.png

 

Commit the config.

 

Summary

Using MineMeld is a powerful and easy way to bring in 3rd party threat feeds based on IP, URL, and Domain. Using these feeds in your security policy is as easy as pointing the firewall to the published list and referring to the list in a policy. There are many use cases for EDL’s in both positive and negative enforcement scenarios. See the Live link below for additional ideas on incorporating EDL’s with MineMeld into your enterprise security operations.

 

To learn more about the free MineMeld tool:

https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld

 

To learn more about External Dynamic Lists:

https://www.paloaltonetworks.com/documentation/81/pan-os/pan-os/policy/use-an-external-dynamic-list-...

 

Comments

Thanks and congratulations @kwall00 for such a great article.

 

I'd like to take the chance to point two additional resources:

@xhoms appreciate the additional information! 

@xhoms @arsimon The above is great however it doesn't work with a taxiiclient class prototype, is there a way we can do this with that type of prototype?

Ask Questions Get Answers Join the Live Community
Article Dashboard
Version history
Revision #:
4 of 4
Last update:
‎08-23-2018 08:37 AM
Updated by:
 
Contributors