- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
on 09-04-2017 05:19 PM - edited on 12-14-2021 06:04 AM by jforsythe
Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.
MineMeld was conceived as a Threat Indicator Sharing platform. Its main goal is as a tool to help users consume all sorts of threat indicators from diverse sources, aggregate them, age them, then finally make them actionable by featuring very flexible output nodes. Over time, MineMeld has incorporated Data API's and local storage for indicators, which has extended its range of use cases. In this article in the Using MineMeld... series, we'll cover the use case of MineMeld as a Incident Response Platform.
The following bullets describe the need for Incident Response we'll solve in this article:
First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated to the latest version of Minemeld.
Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)
At the end of this article you'll find copy&paste ready snipets. But we highly encourage you to follow the steps to get used to MineMeld's WEB UI features and procedures.
Let's start with a blank configuration. The following is the summary of steps we'll take:
To start we must reach the prototype library that is available by navigating to small icon in the down left part of the Config Tab.
In this example we'll trust all indicators coming from VirusTotal and WildFire. This means that any SHA256 hash delivered by these miners will have a value of 100 in the confidence attribute. So we can just clone the stdlib.feedHCGreenWithValue prototype that will accept all indicators with confidence > 75 shared in the green level.
We can, as well, clone the stdlib.aggregatorSHA256 from the prototype library into our graph.
The VirusTotal miner prototype in the library is still tagged as experimental which means that all indicators used by it will be placed in the red share level. As our output node only accepts indicators in the green share level, we must create a new prototype using the one in the library as a base but chaging the share level attribute value.
Note that the prototype in the library shares in the red channel. Click on "new" to create a new prototype that shares in the green level instead.
Now it is time to clone this recently created prototype into our graph.
We, finally, need a miner node that could receive async events (WildFire logs from a PANOS device) and store them into a local database. This miner must also provide aging for the indicators so they are removed from the list at a given expiration time. This is what the class minemeld.ft.localdb.Miner exposed through the stdlib.localDB prototype is all about.
As in the case of the VirusTotal prototype, this node is tagged as experimental which means we must first create a new prototype to share indicators in the green level before we can clone it into our graph.
At this moment our configuration should look like the one in the following picture. Note that the nodes are not linked. We can bind them together by clicking on the input cell of both the aggregator and the output node.
Once all nodes are conneted we just need to commit the configuration to get our working graph.
The graph can be shown by clicking on any node in the Nodes Tab and selecting the graph view.
A good practice is to use authentication in the output nodes so only approved clients can grab indicators from it. MineMeld features a very flexible feed authentication mechanism: The administrator can create "feed users" that can latter be attached as "tags" to the output nodes. A given output node can host multiple tags which means it will accepts connections from more that one customer.
First of all let's access the feed user panel available in the Admin main tab. Once there, a new feed user can be added by clicking on the plus icon.
The feed user must then be attached to the output node. That can be achieved by clicking on the output node from the Nodes main Tab. In the node's status pannel you must click on the TAG row to attach the feed user authentication to that node.
We are all set for now. We just need the miners to start gathering indicators for us.
You might have noticed the small error icon besides the VirusTotal miner node. That error indicates that the node is not fully configured. A mandatory attribute is not yet provided: The VirusTotal Intelligece API Key.
The key can be obained from the VirusTotal page under the profile settings. Note, please, that the API Key must come from a Virus Total Account with Intelligence access permissions. If you're not familiar with VirusTotal malware hunting features please visit https://www.virustotal.com/intelligence/help/malware-hunting/ before using the VT miner.
Once you have copied the key to your clipboard just paste it into the API KEY field available in the status tab of the virustotal node panel.
No commit is needed. Just force a polling by clicking on the icon at the end right of the LAST RUN field. If everything is right the indicators field will be updated with a non zero value. If you want to take a look to these indicators just navigate to the output node an click on the FEED BASE URL to get a plain text version of the list.
The last step in our mission today is to capture WildFire events from our PANOS device to extract SHA256 indicators out of them. To achieve this we'll leverage the HTTP Forwarding feature introduced with PANOS 8.0.
The following is our theory of operations:
First of all we need a new HTTP Server Profile for the MineMeld. We must provide:
The localDB miners's API accepts payloads either in JSON or PlainText format. In any case the type and indicator fields are the only mandatory ones. Optionals fields are:
A couple of comments in the payload format shown in the picture above:
The next step is to create a new log forwarding object to feature this collection of HTTP server and payload format we've just defined.
Finally we must attach this log forwarding object to the security policy rules that features WildFire inspection.
If you do not want to wait for WildFire incidents just trigger a new one by downloading the WildFire test PE file. Once you get your incident just confirm that it is available in the MineMeld's output feed.
MineMeld's feed output node, by default, provides the content in a plain text format. But you can modify this behavior with additional attributes as explained in Parameters for the output node to easy the consumption by third parties. And, why not, contribute to the MineMeld OpenSource project with new output nodes like, McAfee DXL (TIE) or Tanium.
Local Prototypes used in this use case (/opt/minemeld/local/prototypes/minemeldlocal.yml)
author: minemeld-web description: Local prototype library managed via MineMeld WebUI prototypes: localDBGreen: class: minemeld.ft.localdb.Miner config: attributes: confidence: 100 share_level: green interval: 3600 description: 'list of indicators. Use a name starting with "wl" to create a whitelist for an aggregator ' development_status: EXPERIMENTAL indicator_types: - any node_type: miner tags: - ConfidenceAny - ShareLevelAny virusTotalShareGreen: class: minemeld.ft.vt.Notifications config: age_out: default: 365d interval: 7200 sudden_death: false attributes: confidence: 100 share_level: green description: 'Miner for VirusTotal Intelligence Notifications feed. ' development_status: EXPERIMENTAL indicator_types: - md5 - sha256 - sha1 node_type: miner tags: - ConfidenceHigh - ShareLevelRed
Graph configuration (Config -> Import)
nodes: VirusTotal: inputs: [] output: true prototype: minemeldlocal.virusTotalShareGreen WildFireEvent: inputs: [] output: true prototype: minemeldlocal.localDBGreen sha256agg: indicator_types: - sha256 inputs: - VirusTotal - WildFireEvent node_type: processor output: true prototype: stdlib.aggregatorSHA256 sha256out: indicator_types: - any inputs: - sha256agg node_type: output output: false prototype: stdlib.feedHCGreenWithValue
In MineMeld 0.9.42 we have introduced a new type of Miner (stdlib.localDB) exposing the HTTP-based API used in this article. In the following examples we assume that you have created a localDB Miner called FooBar.
You can push new indicators using JSON. The ttl attribute is the expiration time in seconds of the indicator. If omitted the default config value of the Miner will be used. Only the indicator and type attributes are mandatory.
curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
{
"indicator": "8.8.8.8",
"type": "IPv4",
"comment": "usual Google DNS Public IP",
"share_level": "green",
"confidence": 100,
"ttl": 3600
}'
You can use the same API call to delete an indicator, by setting the ttl attribute to 0:
curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
{
"indicator": "8.8.8.8",
"type": "IPv4",
"ttl": 0
}'
You can also use a simple text format to push a list of indicators:
Example:
curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
URL
https://www.google.com
comment
test comment
confidence
80
share_level
green
ttl
1800
IPv4
8.8.8.8
comment
usual Google DNS Public IP
share_level
green
confidence
100
ttl
86400
'
You can use the same API call to delete an indicator, by setting the ttl attribute to 0:
curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
URL
https://www.google.com
ttl
0
IPv4
8.8.8.8
ttl
0
'
I found error on Virutotal Miner
2017-09-06T10:40:12 (2906)basepoller._polling_loop INFO: Polling virustotal
2017-09-06T10:40:12 (2906)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.virustotal.com
2017-09-06T10:40:13 (2906)basepoller._poll ERROR: Exception in polling loop for virustotal: No JSON object could be decoded
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/vt.py", line 88, in _build_iterator
return super(Notifications, self)._build_iterator(now)
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 135, in _build_iterator
result = self.extractor.search(r.json())
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/requests/models.py", line 819, in json
return json.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2017-09-06T10:40:17 (2906)basepoller._actor_loop INFO: virustotal - command: 1504669206980 age_out
2017-09-06T10:40:17 (2906)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-09-06T10:40:17 (2906)basepoller._actor_loop INFO: virustotal - command: 1504669206980 gc
2017-09-06T10:40:17 (2906)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2017-09-06T10:40:22 (2906)basepoller._actor_loop INFO: dshield_blocklist - command: 1504669222341 age_out
2017-09-06T10:40:22 (2906)table._query_by_index INFO: Deleted in scan of _age_out: 0
Please advise me
Thank you
@iThreatHunt : Could you, please, explore the following URL in your browser?
https://www.virustotal.com/intelligence/hunting/notifications-feed/?key=<your_vt_api_key>
Looks like it is returning a blank document instead of a json one. If that's the case then you should, as stated at the begining of the article:
The VT Miner connects to the Virus Total Hunting Notification service to grab incidents that match your YARA rule. If you don't complete these pre-steps then the VT Miner will receive a blank document an rise that exception (working as expected)
@HAO.BAN, it was introduced in MineMeld 0.9.42. Instructions on how to upgrade for Ubuntu distributions at https://live.paloaltonetworks.com/t5/MineMeld-Articles/Updating-MineMeld/ta-p/72151
@xhomsThanks! I followed this article https://live.paloaltonetworks.com/t5/MineMeld-Discussions/What-s-new-in-MineMeld-0-9-18/m-p/98409/hi... and run
$ /usr/sbin/minemeld-auto-update
the version is 0.9.44 now.
@iThreatHunt : AFAIK, to push an indicator to McAfee's TIE through DXL you need to own the SHA256, SHA1 and MD5 hashes of the samples. That makes the process a bit more complicated as you must correlate the SHA256 value before pushing to TIE. Working on that though.
@xhoms How do you config on MM & McAfee Product? Could you recommend to me?
I 'm staritng to learning about McAfee Threat Intelligence Exchange (TIE) DXL.
Thank you.
@iThreatHunt : A Couple of resources
The output node is not receiving any indicators from the aggregator. The aggregator is showing DROP_UPDATE for each event received from wildfire.
Hi @xhoms
Below is the prototype config.
infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- type == 'sha256'
name: accept SHA256
- actions:
- drop
name: drop all
whitelist_prefixes:
- wl
@AlexFoPE , I'm afraid you can't. You should interface directly with WildFire to get other hash values (SHA1, MD5) for that given sample. Have you taken a look to the XSOAR Threat Intel Management? That's a whole new dimension regards threat intel management.