Uploading list of indicators to MineMeld

Printer Friendly Page

Local indicators

MineMeld supports Miners where the list of indicators is stored in a local database inside the MineMeld instance. These Miners can be used to define a static list of malicious indicators or a static whitelist (for more details about using a Miner as a whitelist check the article https://live.paloaltonetworks.com/t5/MineMeld-Articles/Creating-whitelists/ta-p/72250).

 

You can add, delete and change the indicators stored in these Miners directly from the WebUI (the following screenshot shows an example of a whitelist, same applies to malicious indicator lists):

Screen Shot 2017-06-20 at 12.50.37.png

 

Prototypes for Miners of local indicators

stdlib.listDomainGeneric

List of domain indicators
stdlib.listIPv4Generic List of IPv4 indicators
stdlib.listIPv6Generic List of IPv6 indicators
stdlib.listURLGeneric List of URL indicators

 

Uploading indicators to MineMeld

Using the MineMeld API you can upload indicators to Miners of local indicators. This can be automated using the minemeld-sync.py script: https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785

 

Requirements

To use the script Python 2.7.9+ is required. If you are planning to use the script from Linux or Mac OS X, python should be already available in your environment. Otherwise you can download and install python from here https://www.python.org

 

List of indicators

The list of indicators to be uploaded should be stored in a plain text file, one line per indicator:

8.8.8.8
8.8.4.4
10.0.0.0/8

You can also add comments to each indicator, to be stored in the indicator comment attribute in MineMeld:

# Google public DNS (this will be placed in the comment attribute)
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
8.8.4.4
# Private network (in the comment attribute, again)
10.0.0.0/8

You can also specify custom attributes, with the format @<attribute name>: <attribute value>:

# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.4.4
# Private network (in the comment attribute, again)
# @direction: inbound
# @confidence: 60
10.0.0.0/8

 

How to

  1. Download the script from this webpage and save it in a file called minemeld-sync.py. If you have wget installed you can use:
    wget -O minemeld-sync.py https://gist.githubusercontent.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785/raw
  2. Use the following command line to upload the IPv4 indicators stored in the file indicators.lst to the MineMeld instance with URL https://192.168.1.1, using the username admin and password minemeld:
    python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
  3. (OPTIONAL) By default old indicators are not removed from the Miner. If you prefer a full sync of the local list with the remote miner, add the options --update and --delete to the command line:

     

    python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --delete --update --share-level red wlWhiteListIPv4 indicators.lst

 

Remote certificate verification

By default remote MineMeld certificate is verified using certifi package (if installed), or using the CA bundle file or CA certs directory specified via the --ca-path option:


python minemeld-sync.py -m https://192.168.1.1 --ca-path /etc/ssl/certs -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst

 

To disable remote certificate verification use the option -k:


python minemeld-sync.py -m https://192.168.1.1 -k -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst

 

Comments

Thank you for this Imori!

 Comment field in the script would be much appreciated Luigi.  I like to mark why I am blacklisting an IP and have record of what it was weeks later when I am looking at the IP Addresses as a reminder. Possible?

 

I am doing 632 comments manually right now in the GUI after running the script to add all those IP addresses to my Miner. 

Hi @Kyle_Buffington,

absolutely, that's something I wanted to add. About the syntax, would something like this work ?

# comment for indicator1
indicator1
# comment for indicator2
indicator2
...

Example

# Google Public DNS 1
8.8.8.8
# Google Public DNS 2
8.8.4.4

Hello,

Following up on the previous comment, I think it will be useful to have comment, confidence and share level as part of the file. I tweaked your script to take as input a csv file with those fields. Code below.

 

 import csv

def _merge_lists(file_paths):
    result = {}
    for l in file_paths:
        with open(l) as csvfile:
            ioclist = csv.DictReader(csvfile)
            for row in ioclist:
                value['share_level'] = row['share_level']
                value['confidence'] = row['Confidence']
                value['comment'] = row['Description'] +'. '+ row['Source'] +'. ' + row['Tags']
                result[row['Value']] = value
    return result

The challenge I have is that I need to import about 20k IOC. I get an HTTP error 413 when I get to about 3700. Any idea how I could solve the issue?

 

the script does not seems to be working  when i use python3.7, any idea on what's need to be changed in the script to get it working on 3.7?

 

Hello Imori,

I get the following errors when running your script. Do you know what cause it ?  I jus upgrade to python 2.7.16

 

WARNING:__main__:MineMeld cert verification disabled
Traceback (most recent call last):
File "minemeld-sync.py", line 489, in <module>
main()
File "minemeld-sync.py", line 441, in main
if not MM.check(miner=args.miner, type_=args.type):
File "minemeld-sync.py", line 186, in check
content = self._call_api('/status/minemeld')
File "minemeld-sync.py", line 178, in _call_api
context=self.context
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: UNAUTHORIZED

Hello Imori,

When I use python3.4.3, I get the following error.

 

hguo1@MineMeld:~$ python3 minemeld-sync.py -m https://10.181.44.13 -k -u lo_hguo1 -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
File "minemeld-sync.py", line 209
except urllib2.HTTPError, e:

Awesome, thanks Lmori!

Python2.7 env worked as expected :)

Ask Questions Get Answers Join the Live Community
Version history
Revision #:
6 of 6
Last update:
‎07-14-2017 01:59 AM
Updated by:
 
Contributors