- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
12-04-2019 12:16 PM - edited 12-04-2019 12:32 PM
Minemeld is very flexible and very useful for manipulating indicator lists which are then available to Palo Alto Networks Firewalls for ingestion. The challenge presented to me was how to incorporate this capability into the existing customer workflow so that internal manual processes could be avoided and replaced in favor of this automated pipeline.
This is the POC for that challenge, hopefully you find this useful.
This is what we are going to build
There will be a local script that will push indicators from an existing workflow to Minemeld (hosted in AF), there will be by default no expiration on the indicators except for what is provided by the customers pipeline, there will be a safe list of static IP's (in this case) that will be whitelisted so that the customer can't block themselves or important IP addresses and then an output node to make the indicators available to firewalls and other tools.
Build the Minemeld pipeline.
Step 1 - Custom Miner for stdlib.localdb
We need to create a custom stdlib.localdb miner to account for a few things that are not set by default.
expiry - default expiry is 24 hours
tlp - red is the default, need to make it green
Clone the stdlib.localdb > find it in the Prototypes section and click New
Change the config that is presented to this
age_out:
sudden_death: false
default: null
interval: 60
default_ttl: 63072000
attributes:
confidence: 100
share_level: green
then give it a new name - I called it stdlib_local_perp_green (ts actually a 1 year expiry but long enough for the purposes of the POC)
Now we can create the nodes we need using this.
Go to config, click the eye at the bottom and click the (+) to create a new node
Add a new node > name it in this case (api_upload_stdlib_localdb_perpetual_ipv4)
select the prototype of the item we just created and click Ok
this will be where we will push the indicators to from the pipeline
Create the Processor node
Again click the eye, click the (+) and create a new stdlib.aggregatorIPv4Generic and link it to the previously created node
give it a name and click ok
Create the Output node
Again click the eye, click the (+) and create a new stdlib.feedHCGreenWithValue and link it to the previously created processor node
give it a name and click ok
last item is the safe list
Again click the eye, click the (+) and create a new stdlib.listIPv4Generic miner
Give it a name and click ok (make sure the name begins with wl so that its marked as whitelist and removes the entries in its list from the attached processor node)
Now link the safe list to the processor node buy clicking the config tab then locate the processor we created and click in the inputs column to pop up an editor where we add the safe list static miner.
Now we should see two entries in that processor showing the 2 miners.
Click commit and wait for the services to come up and we should now have the picture at the top.
Now we need the script to push indicators, thankfully we have a good prototype here for minemeld_sync.py which we can use.
https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785
We just need to figure out how to find the url to upload to, set the credentials and test.
Under Admin, create a new user and password to use for uploading via the script
Locate the location of the output node address, we will use that to figure out the upload url.
click Nodes > locate the output node and click on it and locate the Feed base URL
https://<x>.<palo domain>.com/feeds/<name of output node>
Capture the <x>.<palo domain>.com
download and save the minemeld_sync.py script
create a input file to upload sample indicators (mine is called input.json)
[
{
"indicator": "5.5.5.5",
"value": {
"comment": "15 min expiry",
"confidence": 100,
"type": "IPv4",
"share_level": "green",
"ttl": 900
}
},
{
"indicator": "6.6.6.6",
"value": {
"comment": "30 min expiry",
"confidence": 100,
"type": "IPv4",
"share_level": "green",
"ttl": 1800
}
},
{
"indicator": "7.7.7.7",
"value": {
"comment": "no expire",
"confidence": 100,
"type": "IPv4",
"share_level": "green"
}
}
]
this will upload 3 indicators, 2 that have expiry set and one that will have the default expiry of 1 year
now we can run the minemeld_sync script
./minemeld-sync.py -m https://<x>.<palo domain>.com -k -u <user> -p '<password>' -t IPv4 api_upload_stdlib_localdb_perpetual_ipv4 input.json
This should give you output like this (works for python 2.7 has issues with python3)
WARNING:__main__:MineMeld cert verification disabled
INFO:__main__:A#0 - 5.5.5.5 (add)
INFO:__main__:A#1 - 6.6.6.6 (add)
INFO:__main__:A#2 - 7.7.7.7 (add)
INFO:__main__:Done
check the miner to see that the indicators are there
Nodes > locate the miner and click on the table indicator
We can see the 2 indicators with expiration times and one with none (default for miner)
Lets check the output node to see that they are pushed all the way through
Click Nodes > locate the output node and click the feedbase URL to open the page that shows the contents of the output node
You should see
5.5.5.5-5.5.5.5
6.6.6.6-6.6.6.6
7.7.7.7-7.7.7.7
Now update the safe list miner to exclude 6.6.6.6
Nodes > locate the safe list miner and click the table icon
click (+) and add 6.6.6.6 (uncheck red share level) and click ok
Now refresh the output miner page and you should see
5.5.5.5-5.5.5.5
7.7.7.7-7.7.7.7
Now you can configure your infrastructure to connect to the output node and pull down the indicators that have been built into the indicator pipeline and expire as required and set in the JSON file (which can be output from the existing intel platform programmatically).
Remember if required to set the feed user and attach it to the output node via the nodes > output > click tags and add the tag related to the feed user created (and tag created there)
thanks to @lmori for help on this
Eric
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!