TI Automation - Indicator Pipeline with Safe List

Showing results for 
Show  only  | Search instead for 
Did you mean: 

TI Automation - Indicator Pipeline with Safe List

L1 Bithead

Minemeld is very flexible and very useful for manipulating indicator lists which are then available to Palo Alto Networks Firewalls for ingestion.  The challenge presented to me was how to incorporate this capability into the existing customer workflow so that internal manual processes could be avoided and replaced in favor of this automated pipeline.  


This is the POC for that challenge, hopefully you find this useful.


This is what we are going to build




There will be a local script that will push indicators from an existing workflow to Minemeld (hosted in AF), there will be by default no expiration on the indicators except for what is provided by the customers pipeline, there will be a safe list of static IP's (in this case) that will be whitelisted so that the customer can't block themselves or important IP addresses and then an output node to make the indicators available to firewalls and other tools.


Build the Minemeld pipeline.

Step 1 - Custom Miner for stdlib.localdb

We need to create a custom stdlib.localdb miner to account for a few things that are not set by default.

expiry - default expiry is 24 hours

tlp - red is the default, need to make it green


Clone the stdlib.localdb > find it in the Prototypes section and click New


Change the config that is presented to this


    sudden_death: false
    default: null
    interval: 60
default_ttl: 63072000
    confidence: 100
    share_level: green



then give it a new name - I called it stdlib_local_perp_green (ts actually a 1 year expiry but long enough for the purposes of the POC)


Now we can create the nodes we need using this.


Go to config, click the eye at the bottom and click the (+) to create a new node

Add a new node > name it in this case (api_upload_stdlib_localdb_perpetual_ipv4)


select the prototype of the item we just created and click Ok

this will be where we will push the indicators to from the pipeline


Create the Processor node

Again click the eye, click the (+) and create a new stdlib.aggregatorIPv4Generic and link it to the previously created node

give it a name and click ok


Create the Output node

Again click the eye, click the (+) and create a new stdlib.feedHCGreenWithValue and link it to the previously created processor node

give it a name and click ok


last item is the safe list

Again click the eye, click the (+) and create a new stdlib.listIPv4Generic miner

Give it a name and click ok (make sure the name begins with wl so that its marked as whitelist and removes the entries in its list from the attached processor node)


Now link the safe list to the processor node buy clicking the config tab then locate the processor we created and click in the inputs column to pop up an editor where we add the safe list static miner.


Now we should see two entries in that processor showing the 2 miners.


Click commit and wait for the services to come up and we should now have the picture at the top.


Now we need the script to push indicators, thankfully we have a good prototype here for minemeld_sync.py which we can use.



We just need to figure out how to find the url to upload to, set the credentials and test.


Under Admin, create a new user and password to use for uploading via the script


Locate the location of the output node address, we will use that to figure out the upload url.

click Nodes > locate the output node and click on it and locate the Feed base URL


https://<x>.<palo domain>.com/feeds/<name of output node>


Capture the <x>.<palo domain>.com 


download and save the minemeld_sync.py script

create a input file to upload sample indicators (mine is called input.json)


        "indicator": "",
        "value": {
            "comment": "15 min expiry",
            "confidence": 100,
            "type": "IPv4",
            "share_level": "green",
            "ttl": 900   
        "indicator": "",
        "value": {
            "comment": "30 min expiry",
            "confidence": 100,
            "type": "IPv4",
            "share_level": "green",
            "ttl": 1800
        "indicator": "",
        "value": {
            "comment": "no expire",
            "confidence": 100,
            "type": "IPv4",
            "share_level": "green"



this will upload 3 indicators, 2 that have expiry set and one that will have the default expiry of 1 year


now we can run the minemeld_sync script


./minemeld-sync.py -m https://<x>.<palo domain>.com -k -u <user> -p '<password>' -t IPv4 api_upload_stdlib_localdb_perpetual_ipv4 input.json


This should give you output like this (works for python 2.7 has issues with python3)


WARNING:__main__:MineMeld cert verification disabled
INFO:__main__:A#0 - (add)
INFO:__main__:A#1 - (add)
INFO:__main__:A#2 - (add)


check the miner to see that the indicators are there

Nodes > locate the miner and click on the table indicator


We can see the 2 indicators with expiration times and one with none (default for miner)


Lets check the output node to see that they are pushed all the way through

Click Nodes > locate the output node and click the feedbase URL to open the page that shows the contents of the output node

You should see


Now update the safe list miner to exclude

Nodes > locate the safe list miner and click the table icon

click (+) and add (uncheck red share level) and click ok

Now refresh the output miner page and you should see



Now you can configure your infrastructure to connect to the output node and pull down the indicators that have been built into the indicator pipeline and expire as required and set in the JSON file (which can be output from the existing intel platform programmatically).


Remember if required to set the feed user and attach it to the output node via the nodes > output > click tags and add the tag related to the feed user created (and tag created there)


thanks to @lmori  for help on this








Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!