Using MineMeld as an Incident Response Platform

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
L5 Sessionator
No ratings

Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.


MineMeld was conceived as a Threat Indicator Sharing platform. Its main goal is as  a tool to help users consume all sorts of threat indicators from diverse sources, aggregate them, age them, then finally make them actionable by featuring very flexible output nodes. Over time, MineMeld has incorporated Data API's and local storage for indicators, which has extended its range of use cases. In this article in the Using MineMeld... series, we'll cover the use case of MineMeld as a Incident Response Platform.

 

The following bullets describe the need for Incident Response we'll solve in this article:

  • You own a VirusTotal Intelligence account that you use to perform hunting for malware samples with YARA rules. The corresponding VirusTotal notification service provides you with fresh malware incidents (SHA256 hashes) seen on its community of users that matches the YARA rules you've provided.
  • You have deployed PA-Series Next Generation Firewalls for segmentation (either Internet Gateway or Datacenter) with WildFire subscription that unmask unknown malware crossing your network (incident) in less than 5 minutes.
  • A set of endpoints is secured with TRAPS (effectively blocking zero day malware). But you also take care of a large base of endpoints protected by legacy AV solutions that do not provide protections against zero day malware. But that same AV solution can block execution of binaries provided someone could keep a list of SHA256 hashes belonging to malicious files that should be blacklisted.
  • So you want MineMeld to provide a response keeping an updated output node with malware hashes coming both from VT and WildFire incidents that will be consumed by the legagy AV solution. This way you buy time to execute the migration of the large installed base to TRAPS.

Step 1. Deploy MineMeld

First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.

 

Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)

 

Step 2. Create the MineMeld Graph

At the end of this article you'll find copy&paste ready snipets. But we highly encourage you to follow the steps to get used to MineMeld's WEB UI features and procedures.

 

Let's start with a blank configuration. The following is the summary of steps we'll take:

  1. Clone the out-of-the-box stdlib.feedHCGreenWithValue prototype into our graph as output node.
  2. Clone the out-of-the-box stdlib.aggregatorSHA256 prototype into our graph as aggregator node.
  3. Create a new prototype based on the virustotal.notifications one to be able to share VT intelligence into the green share level.
  4. Create a new prototype based on the stdlib.localDB one to be able to store WildFire events into the green share level.
  5. Link the nodes to form a connected graph. 

To start we must reach the prototype library that is available by navigating to small icon in the down left part of the Config Tab.

 

Picture01_1.png

 

In this example we'll trust all indicators coming from VirusTotal and WildFire. This means that any SHA256 hash delivered by these miners will have a value of 100 in the confidence attribute. So we can just clone the stdlib.feedHCGreenWithValue prototype that will accept all indicators with confidence > 75 shared in the green level.

 

Picture02_1.png

 

Picture03_1.png

 

Picture04_1.png

 

We can, as well, clone the stdlib.aggregatorSHA256 from the prototype library into our graph.

 

Picture05_1.png

 

Picture06_1.png

 

Picture07_1.png

 

The VirusTotal miner prototype in the library is still tagged as experimental which means that all indicators used by it will be placed in the red share level. As our output node only accepts indicators in the green share level, we must create a new prototype using the one in the library as a base but chaging the share level attribute value.

 

Picture08_1.png

 

Picture09_1.png

 

Note that the prototype in the library shares in the red channel. Click on "new" to create a new prototype that shares in the green level instead.

 

Picture10_1.png

 

Now it is time to clone this recently created prototype into our graph.

 

Picture11_1.png

 

Picture12_1.png

 

Picture13_1.png

 

We, finally, need a miner node that could receive async events (WildFire logs from a PANOS device) and store them into a local database. This miner must also provide aging for the indicators so they are removed from the list at a given expiration time. This is what the class minemeld.ft.localdb.Miner exposed through the stdlib.localDB prototype is all about.

 

As in the case of the VirusTotal prototype, this node is tagged as experimental which means we must first create a new prototype to share indicators in the green level before we can clone it into our graph.

 

Picture14_1.png

 

Picture15_1.png

 

Picture16_1.png

 

Picture17_1.png

 

Picture18_!.png

 

Picture19_1.png

 

At this moment our configuration should look like the one in the following picture. Note that the nodes are not linked. We can bind them together by clicking on the input cell of both the aggregator and the output node.

 

Picture20_1.png

 

Once all nodes are conneted we just need to commit the configuration to get our working graph.

 

Picture21_1.png

 

The graph can be shown by clicking on any node in the Nodes Tab and selecting the graph view. 

 

Picture26.png

 

Step 3. Configure authentication in the output node

A good practice is to use authentication in the output nodes so only approved clients can grab indicators from it. MineMeld features a very flexible feed authentication mechanism: The administrator can create "feed users" that can latter be attached as "tags" to the output nodes. A given output node can host multiple tags which means it will accepts connections from more that one customer.

 

First of all let's access the feed user panel available in the Admin main tab. Once there, a new feed user can be added by clicking on the plus icon.

 

Picture21_1_feedUser.png  

Picture21_1_feedUser_2.pngPicture21_1_feedUser_3.png

 

The feed user must then be attached to the output node. That can be achieved by clicking on the output node from the Nodes main Tab. In the node's status pannel you must click on the TAG row to attach the feed user authentication to that node.

 

 

 

Picture21_1_feedUser_4.png

 

Picture21_1_feedUser_5.png

 

 

 

We are all set for now. We just need the miners to start gathering indicators for us.

 

Step 4. Connect MineMeld with VirusTotal

You might have noticed the small error icon besides the VirusTotal miner node. That error indicates that the node is not fully configured. A mandatory attribute is not yet provided: The VirusTotal Intelligece API Key.

 

The key can be obained from the VirusTotal page under the profile settings. Note, please, that the API Key must come from a Virus Total Account with Intelligence access permissions. If you're not familiar with VirusTotal malware hunting features please visit https://www.virustotal.com/intelligence/help/malware-hunting/ before using the VT miner.

 

Picture24_1.png

 

Picture25_1.png

 

Once you have copied the key to your clipboard just paste it into the API KEY field available in the status tab of the virustotal node panel.

 

Picture22_!.png

 

Picture23_1.png

 

No commit is needed. Just force a polling by clicking on the icon at the end right of the LAST RUN field. If everything is right the indicators field will be updated with a non zero value. If you want to take a look to these indicators just navigate to the output node an click on the FEED BASE URL to get a plain text version of the list.

 

Picture32_1.png

 

Picture33.png

Step 5. Configure WildFire incident forwarding

The last step in our mission today is to capture WildFire events from our PANOS device to extract SHA256 indicators out of them. To achieve this we'll leverage the HTTP Forwarding feature introduced with PANOS 8.0.

 

The following is our theory of operations:

  • PANOS generates a WildFire event log with multiple fields. One of them is the "file_digest" that contains the SHA256 of the recently detected malware.
  • A log forwarding object is attached to the security rule that features WildFire inspection
  • The log forwarding object accepts all events (no filter needed) in the WildFire category, formats them in a way that the miner node can consume and forwards it using HTTPS to MineMeld.
  • MineMeld's default distribution includes a self signed certificate. For the HTTPS protocol to work you must:
    1. Issue a certificate for MineMeld's web server from a PKI that is in the trust list of the PANOS Device. Alternatively you can use the PANOS Device as a PKI and generate the MineMeld certificate from it.
    2. Follow the instructions in https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-SSL-Certificates/m-p/112938 to import that certificate into your MineMeld instance.

First of all we need a new HTTP Server Profile for the MineMeld. We must provide:

  • A name
  • An address or FQDN
  • Protocol and Port
  • Method
  • Credentials (we can use any MineMeld admin user credential. Feed user credentials won't work)

 

Picture34_1.png

 

The localDB miners's API accepts payloads either in JSON or PlainText format. In any case the type and indicator fields are the only mandatory ones. Optionals fields are:

  • share_level: to override miner's share_level value. A given threat intel source might want to specify the share level of a given indicator.
  • ttl: the amount of seconds the indicators should be kept into the store. A null ttl value means the indicator won't ever expire. In this example we'll use a 1h ttl so you can verify how the indicators are aged out by the miner.
  • any other attribute will just be attached to the indicator to enrich it.

Picture35_1.png

 

A couple of comments in the payload format shown in the picture above:

  • The url format must be that exact literal just changing the text in red to match the miner's node name we want to forward the indicators to. In our example /config/data/WildFireEvent_indicators/append
  • A Content-Type header with the value application/json is needed for the MM API to accept json data.
  • The parameter t must be equal to localdb to forward the received data to a localdb type of miner.
  • The optional parameter h can contain a node name. If it exists the MineMeld engine will trigger a node "sync request" to the named node after the data has been processed. In this case we want the indicator captured to be automatically flushed down the graph path so we request a sync of the WildFireEvent node.
  • Note the $filedigest variable in the payload field. This tells the PANOS forwarding engine to replace that space in the payload with the corresponding field value in the event log.

The next step is to create a new log forwarding object to feature this collection of HTTP server and payload format we've just defined.

 

Picture36_1.pngPicture36_2.png

 

Finally we must attach this log forwarding object to the security policy rules that features WildFire inspection.

 

Picture37_2.png

 

If you do not want to wait for WildFire incidents just trigger a new one by downloading the WildFire test PE file. Once you get your incident just confirm that it is available in the MineMeld's output feed.

 

Picture38_1.png

 

 

 

 

 

Picture41_1.png

 

MineMeld's feed output node, by default, provides the content in a plain text format. But you can modify this behavior with additional attributes as explained in Parameters for the output node to easy the consumption by third parties. And, why not, contribute to the MineMeld OpenSource project with new output nodes like, McAfee DXL (TIE) or Tanium.

 

Annex 1: the prototypes

Local Prototypes used in this use case (/opt/minemeld/local/prototypes/minemeldlocal.yml)

 

author: minemeld-web
description: Local prototype library managed via MineMeld WebUI
prototypes:
    localDBGreen:
        class: minemeld.ft.localdb.Miner
        config:
            attributes:
                confidence: 100
                share_level: green
            interval: 3600
        description: 'list of indicators. Use a name starting with "wl" to create
            a whitelist for an aggregator

            '
        development_status: EXPERIMENTAL
        indicator_types:
        - any
        node_type: miner
        tags:
        - ConfidenceAny
        - ShareLevelAny
    virusTotalShareGreen:
        class: minemeld.ft.vt.Notifications
        config:
            age_out:
                default: 365d
                interval: 7200
                sudden_death: false
            attributes:
                confidence: 100
                share_level: green
        description: 'Miner for VirusTotal Intelligence Notifications feed.

            '
        development_status: EXPERIMENTAL
        indicator_types:
        - md5
        - sha256
        - sha1
        node_type: miner
        tags:
        - ConfidenceHigh
        - ShareLevelRed

 

Graph configuration (Config -> Import)

 

nodes:
  VirusTotal:
    inputs: []
    output: true
    prototype: minemeldlocal.virusTotalShareGreen
  WildFireEvent:
    inputs: []
    output: true
    prototype: minemeldlocal.localDBGreen
  sha256agg:
    indicator_types:
    - sha256
    inputs:
    - VirusTotal
    - WildFireEvent
    node_type: processor
    output: true
    prototype: stdlib.aggregatorSHA256
  sha256out:
    indicator_types:
    - any
    inputs:
    - sha256agg
    node_type: output
    output: false
    prototype: stdlib.feedHCGreenWithValue

Annex 2: the API

In MineMeld 0.9.42 we have introduced a new type of Miner (stdlib.localDB) exposing the HTTP-based API used in this article. In the following examples we assume that you have created a localDB Miner called FooBar.

 

A2.1: Push new indicators via JSON

You can push new indicators using JSON. The ttl attribute is the expiration time in seconds of the indicator. If omitted the default config value of the Miner will be used. Only the indicator and type attributes are mandatory.

curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
{
"indicator": "8.8.8.8",
"type": "IPv4",
"comment": "usual Google DNS Public IP",
"share_level": "green",
"confidence": 100,
"ttl": 3600
}'

 A2.2: Delete indicators via JSON

You can use the same API call to delete an indicator, by setting the ttl attribute to 0:

curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
{
"indicator": "8.8.8.8",
"type": "IPv4",
"ttl": 0
}'

A2.3: Push new indicators via text format

You can also use a simple text format to push a list of indicators:

  • Indicators are divided by a blank line
  • The first line of an indicator is the type of the indicator
  • The second line is the indicator itself
  • After you can specify the attributes, where a line is the attribute name and the line after is the attribute value

Example:

curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
URL
https://www.google.com
comment
test comment
confidence
80
share_level
green
ttl
1800

IPv4
8.8.8.8
comment
usual Google DNS Public IP
share_level
green
confidence
100
ttl
86400
'

A2.4: Delete indicators via text format

You can use the same API call to delete an indicator, by setting the ttl attribute to 0:

curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d '
URL
https://www.google.com
ttl
0

IPv4
8.8.8.8
ttl
0
'

 

Rate this article:
Comments
L3 Networker

I found error on Virutotal Miner

 

2017-09-06T10:40:12 (2906)basepoller._polling_loop INFO: Polling virustotal
2017-09-06T10:40:12 (2906)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.virustotal.com
2017-09-06T10:40:13 (2906)basepoller._poll ERROR: Exception in polling loop for virustotal: No JSON object could be decoded
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/vt.py", line 88, in _build_iterator
return super(Notifications, self)._build_iterator(now)
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 135, in _build_iterator
result = self.extractor.search(r.json())
File "/opt/minemeld/engine/0.9.42/local/lib/python2.7/site-packages/requests/models.py", line 819, in json
return json.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2017-09-06T10:40:17 (2906)basepoller._actor_loop INFO: virustotal - command: 1504669206980 age_out
2017-09-06T10:40:17 (2906)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-09-06T10:40:17 (2906)basepoller._actor_loop INFO: virustotal - command: 1504669206980 gc
2017-09-06T10:40:17 (2906)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2017-09-06T10:40:22 (2906)basepoller._actor_loop INFO: dshield_blocklist - command: 1504669222341 age_out
2017-09-06T10:40:22 (2906)table._query_by_index INFO: Deleted in scan of _age_out: 0

 

Please advise me

 

Thank you

 

L5 Sessionator

@iThreatHunt : Could you, please, explore the following URL in your browser?

https://www.virustotal.com/intelligence/hunting/notifications-feed/?key=<your_vt_api_key>

Looks like it is returning a blank document instead of a json one. If that's the case then you should, as stated at the begining of the article:

  • Verify that your VT account is enabled for Inteligence services (more specifically the "hunting" service)
  • Verify that you've created a YARA rule in VT hunting that describes the incidents you want to track.

The VT Miner connects to the Virus Total Hunting Notification service to grab incidents that match your YARA rule. If you don't complete these pre-steps then the VT Miner will receive a blank document an rise that exception (working as expected)

L2 Linker

In Step 2, why I couldn't find stdlib.localDB? My minemeld version is 0.9.40, Thanks!

L5 Sessionator

@HAO.BAN, it was introduced in MineMeld 0.9.42. Instructions on how to upgrade for Ubuntu distributions at https://live.paloaltonetworks.com/t5/MineMeld-Articles/Updating-MineMeld/ta-p/72151

L2 Linker

@xhomsThanks! I followed this article https://live.paloaltonetworks.com/t5/MineMeld-Discussions/What-s-new-in-MineMeld-0-9-18/m-p/98409/hi... and run

$ /usr/sbin/minemeld-auto-update

the version is 0.9.44 now.

L3 Networker

Hi @xhoms

 

Have you test MineMeld with McAfee DXL (TIE)?

 

Thank you

L5 Sessionator

@iThreatHunt : AFAIK, to push an indicator to McAfee's TIE through DXL you need to own the SHA256, SHA1 and MD5 hashes of the samples. That makes the process a bit more complicated as you must correlate the SHA256 value before pushing to TIE. Working on that though.

L3 Networker

@xhoms How do you config on MM & McAfee Product? Could you recommend to me?

I 'm staritng to learning about McAfee Threat Intelligence Exchange (TIE) DXL.

 

Thank you.

L5 Sessionator

@iThreatHunt : A Couple of resources

L0 Member

The output node is not receiving any indicators from the aggregator. The aggregator is showing DROP_UPDATE for each event received from wildfire.

 

 aggregator-SHA256 DROP_UPDATE "type":sha256", confidence: 100share_level: green"ttl":3600: }sources: ["WildFireEvent"]first_seen: 1514716650029type: {"indicators":"0b29c82b9f87dd2b6924ee9e111c1e9a3d12929bb6d363bb1ce4be34ea559bb4",: "share_level":"green",last_seen: 1514716650029 source_node: WildFireEvent
 aggregator-SHA256 RECVD_UPDATE "type":sha256", confidence: 100share_level: green"ttl":3600: }sources: ["WildFireEvent"]first_seen: 1514716650029type: {"indicators":"0b29c82b9f87dd2b6924ee9e111c1e9a3d12929bb6d363bb1ce4be34ea559bb4",: "share_level":"green",last_seen: 1514716650029 source_node: WildFireEvent
 aggregator-SHA256 DROP_UPDATE "type":sha256", confidence: 100share_level: green"ttl":3600: }sources: ["WildFireEvent"]first_seen: 1514716650029type: {"indicators":"0b29c82b9f87dd2b6924ee9e111c1e9a3d12929bb6d363bb1ce4be34ea559bb4",: "share_level":"green",last_seen: 1514716650029 source_node: WildFireEvent
 aggregator-SHA256 RECVD_UPDATE "type":sha256", confidence: 100share_level: green"ttl":3600: }sources: ["WildFireEvent"]first_seen: 1514716650029type: {"indicators":"0b29c8
 
L5 Sessionator

@virkji, what aggreagator prototype have you used? Are you sure you used the stdlib.aggregatorSHA256 one?

L0 Member

@xhomsyes i used stdlib.aggregatorSHA256 prototype for the aggregator.

L5 Sessionator

Hi @virkji,

 

do you mind sharing your aggregator prototype configuration? Its filter configuration must be dropping the indicators either due to the share_level or aging.

L0 Member

 Hi @xhoms

 

Below is the prototype config.

 

infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- type == 'sha256'
name: accept SHA256
- actions:
- drop
name: drop all
whitelist_prefixes:
- wl

L5 Sessionator

@virkji,

 

could you, please, check the payload format in your PANOS device configuration? (Step 5 -> image). You might have introduced a typo error in the JSON format. It should be:

{
	"type":"sha256",
	"indicator":"$filedigest",
	"share_level":"green",
	"ttl":3600
}

 

L3 Networker

Hello,

 

When i send a test message i got following message how can i solve this issue?

 

/config/data/miner1/append?h=miner1&t=localdb: 401 Unauthorized

L0 Member

@xhoms any way to get sha1 instead of sha256 from paloalto device?

 

 

thank you in advance,

L5 Sessionator

@AlexFoPE , I'm afraid you can't. You should interface directly with WildFire to get other hash values (SHA1, MD5) for that given sample. Have you taken a look to the XSOAR Threat Intel Management? That's a whole new dimension regards threat intel management.

  • 53760 Views
  • 18 comments
  • 2 Likes
Register or Sign-in
Contributors
Labels
Article Dashboard
Version history
Last Updated:
‎12-14-2021 06:04 AM
Updated by: