About epartington

Minemeld is very flexible and very useful for manipulating indicator lists which are then available to Palo Alto Networks Firewalls for ingestion.  The challenge presented to me was how to incorporate this capability into the existing customer workflow so that internal manual processes could be avoided and replaced in favor of this automated pipeline.     This is the POC for that challenge, hopefully you find this useful.   This is what we are going to build     There will be a local script that will push indicators from an existing workflow to Minemeld (hosted in AF), there will be by default no expiration on the indicators except for what is provided by the customers pipeline, there will be a safe list of static IP's (in this case) that will be whitelisted so that the customer can't block themselves or important IP addresses and then an output node to make the indicators available to firewalls and other tools.   Build the Minemeld pipeline. Step 1 - Custom Miner for stdlib.localdb We need to create a custom stdlib.localdb miner to account for a few things that are not set by default. expiry - default expiry is 24 hours tlp - red is the default, need to make it green   Clone the stdlib.localdb > find it in the Prototypes section and click New Change the config that is presented to this   age_out: sudden_death: false default: null interval: 60 default_ttl: 63072000 attributes: confidence: 100 share_level: green     then give it a new name - I called it stdlib_local_perp_green (ts actually a 1 year expiry but long enough for the purposes of the POC) Now we can create the nodes we need using this.   Go to config, click the eye at the bottom and click the (+) to create a new node Add a new node > name it in this case (api_upload_stdlib_localdb_perpetual_ipv4) select the prototype of the item we just created and click Ok this will be where we will push the indicators to from the pipeline   Create the Processor node Again click the eye, click the (+) and create a new stdlib.aggregatorIPv4Generic and link it to the previously created node give it a name and click ok   Create the Output node Again click the eye, click the (+) and create a new stdlib.feedHCGreenWithValue and link it to the previously created processor node give it a name and click ok   last item is the safe list Again click the eye, click the (+) and create a new stdlib.listIPv4Generic miner Give it a name and click ok (make sure the name begins with wl so that its marked as whitelist and removes the entries in its list from the attached processor node)   Now link the safe list to the processor node buy clicking the config tab then locate the processor we created and click in the inputs column to pop up an editor where we add the safe list static miner.   Now we should see two entries in that processor showing the 2 miners.   Click commit and wait for the services to come up and we should now have the picture at the top.   Now we need the script to push indicators, thankfully we have a good prototype here for minemeld_sync.py which we can use. https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785   We just need to figure out how to find the url to upload to, set the credentials and test.   Under Admin, create a new user and password to use for uploading via the script   Locate the location of the output node address, we will use that to figure out the upload url. click Nodes > locate the output node and click on it and locate the Feed base URL   https://<x>.<palo domain>.com/feeds/<name of output node>   Capture the <x>.<palo domain>.com    download and save the minemeld_sync.py script create a input file to upload sample indicators (mine is called input.json)   [ { "indicator": "5.5.5.5", "value": { "comment": "15 min expiry", "confidence": 100, "type": "IPv4", "share_level": "green", "ttl": 900 } }, { "indicator": "6.6.6.6", "value": { "comment": "30 min expiry", "confidence": 100, "type": "IPv4", "share_level": "green", "ttl": 1800 } }, { "indicator": "7.7.7.7", "value": { "comment": "no expire", "confidence": 100, "type": "IPv4", "share_level": "green" } } ]     this will upload 3 indicators, 2 that have expiry set and one that will have the default expiry of 1 year   now we can run the minemeld_sync script   ./minemeld-sync.py -m https://<x>.<palo domain>.com -k -u <user> -p '<password>' -t IPv4 api_upload_stdlib_localdb_perpetual_ipv4 input.json   This should give you output like this (works for python 2.7 has issues with python3)   WARNING:__main__:MineMeld cert verification disabled INFO:__main__:A#0 - 5.5.5.5 (add) INFO:__main__:A#1 - 6.6.6.6 (add) INFO:__main__:A#2 - 7.7.7.7 (add) INFO:__main__:Done   check the miner to see that the indicators are there Nodes > locate the miner and click on the table indicator   We can see the 2 indicators with expiration times and one with none (default for miner)   Lets check the output node to see that they are pushed all the way through Click Nodes > locate the output node and click the feedbase URL to open the page that shows the contents of the output node You should see   5.5.5.5-5.5.5.5 6.6.6.6-6.6.6.6 7.7.7.7-7.7.7.7   Now update the safe list miner to exclude 6.6.6.6 Nodes > locate the safe list miner and click the table icon click (+) and add 6.6.6.6 (uncheck red share level) and click ok Now refresh the output miner page and you should see   5.5.5.5-5.5.5.5 7.7.7.7-7.7.7.7     Now you can configure your infrastructure to connect to the output node and pull down the indicators that have been built into the indicator pipeline and expire as required and set in the JSON file (which can be output from the existing intel platform programmatically).   Remember if required to set the feed user and attach it to the output node via the nodes > output > click tags and add the tag related to the feed user created (and tag created there)   thanks to @lmori  for help on this   Eric           ... View more