set your zone logging profile to enable logging for Zone Protection events to syslog (or other log forwarding methods) https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClHICA0
... View more
Minemeld is very flexible and very useful for manipulating indicator lists which are then available to Palo Alto Networks Firewalls for ingestion. The challenge presented to me was how to incorporate this capability into the existing customer workflow so that internal manual processes could be avoided and replaced in favor of this automated pipeline.
This is the POC for that challenge, hopefully you find this useful.
This is what we are going to build
There will be a local script that will push indicators from an existing workflow to Minemeld (hosted in AF), there will be by default no expiration on the indicators except for what is provided by the customers pipeline, there will be a safe list of static IP's (in this case) that will be whitelisted so that the customer can't block themselves or important IP addresses and then an output node to make the indicators available to firewalls and other tools.
Build the Minemeld pipeline.
Step 1 - Custom Miner for stdlib.localdb
We need to create a custom stdlib.localdb miner to account for a few things that are not set by default.
expiry - default expiry is 24 hours
tlp - red is the default, need to make it green
Clone the stdlib.localdb > find it in the Prototypes section and click New
Change the config that is presented to this
then give it a new name - I called it stdlib_local_perp_green (ts actually a 1 year expiry but long enough for the purposes of the POC)
Now we can create the nodes we need using this.
Go to config, click the eye at the bottom and click the (+) to create a new node
Add a new node > name it in this case (api_upload_stdlib_localdb_perpetual_ipv4)
select the prototype of the item we just created and click Ok
this will be where we will push the indicators to from the pipeline
Create the Processor node
Again click the eye, click the (+) and create a new stdlib.aggregatorIPv4Generic and link it to the previously created node
give it a name and click ok
Create the Output node
Again click the eye, click the (+) and create a new stdlib.feedHCGreenWithValue and link it to the previously created processor node
give it a name and click ok
last item is the safe list
Again click the eye, click the (+) and create a new stdlib.listIPv4Generic miner
Give it a name and click ok (make sure the name begins with wl so that its marked as whitelist and removes the entries in its list from the attached processor node)
Now link the safe list to the processor node buy clicking the config tab then locate the processor we created and click in the inputs column to pop up an editor where we add the safe list static miner.
Now we should see two entries in that processor showing the 2 miners.
Click commit and wait for the services to come up and we should now have the picture at the top.
Now we need the script to push indicators, thankfully we have a good prototype here for minemeld_sync.py which we can use.
We just need to figure out how to find the url to upload to, set the credentials and test.
Under Admin, create a new user and password to use for uploading via the script
Locate the location of the output node address, we will use that to figure out the upload url.
click Nodes > locate the output node and click on it and locate the Feed base URL
https://<x>.<palo domain>.com/feeds/<name of output node>
Capture the <x>.<palo domain>.com
download and save the minemeld_sync.py script
create a input file to upload sample indicators (mine is called input.json)
"comment": "15 min expiry",
"comment": "30 min expiry",
"comment": "no expire",
this will upload 3 indicators, 2 that have expiry set and one that will have the default expiry of 1 year
now we can run the minemeld_sync script
./minemeld-sync.py -m https://<x>.<palo domain>.com -k -u <user> -p '<password>' -t IPv4 api_upload_stdlib_localdb_perpetual_ipv4 input.json
This should give you output like this (works for python 2.7 has issues with python3)
WARNING:__main__:MineMeld cert verification disabled
INFO:__main__:A#0 - 220.127.116.11 (add)
INFO:__main__:A#1 - 18.104.22.168 (add)
INFO:__main__:A#2 - 22.214.171.124 (add)
check the miner to see that the indicators are there
Nodes > locate the miner and click on the table indicator
We can see the 2 indicators with expiration times and one with none (default for miner)
Lets check the output node to see that they are pushed all the way through
Click Nodes > locate the output node and click the feedbase URL to open the page that shows the contents of the output node
You should see
Now update the safe list miner to exclude 126.96.36.199
Nodes > locate the safe list miner and click the table icon
click (+) and add 188.8.131.52 (uncheck red share level) and click ok
Now refresh the output miner page and you should see
Now you can configure your infrastructure to connect to the output node and pull down the indicators that have been built into the indicator pipeline and expire as required and set in the JSON file (which can be output from the existing intel platform programmatically).
Remember if required to set the feed user and attach it to the output node via the nodes > output > click tags and add the tag related to the feed user created (and tag created there)
thanks to @lmori for help on this
... View more