Uploading List of Indicators to MineMeld

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
L7 Applicator
100% helpful (1/1)

Local indicators

MineMeld supports Miners where the list of indicators is stored in a local database inside the MineMeld instance. These Miners can be used to define a static list of malicious indicators or a static whitelist (for more details about using a Miner as a whitelist check the article https://live.paloaltonetworks.com/t5/MineMeld-Articles/Creating-whitelists/ta-p/72250).

 

You can add, delete and change the indicators stored in these Miners directly from the WebUI (the following screenshot shows an example of a whitelist, same applies to malicious indicator lists):

Screen Shot 2017-06-20 at 12.50.37.png

 

Prototypes for Miners of local indicators

stdlib.listDomainGeneric

List of domain indicators
stdlib.listIPv4Generic List of IPv4 indicators
stdlib.listIPv6Generic List of IPv6 indicators
stdlib.listURLGeneric List of URL indicators

 

Uploading indicators to MineMeld

Using the MineMeld API you can upload indicators to Miners of local indicators. This can be automated using the minemeld-sync.py script: https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785

 

Requirements

To use the script Python 2.7.9+ is required. If you are planning to use the script from Linux or Mac OS X, python should be already available in your environment. Otherwise you can download and install python from here https://www.python.org

 

List of indicators

The list of indicators to be uploaded should be stored in a plain text file, one line per indicator:

8.8.8.8
8.8.4.4
10.0.0.0/8

You can also add comments to each indicator, to be stored in the indicator comment attribute in MineMeld:

# Google public DNS (this will be placed in the comment attribute)
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
8.8.4.4
# Private network (in the comment attribute, again)
10.0.0.0/8

You can also specify custom attributes, with the format @<attribute name>: <attribute value>:

# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.4.4
# Private network (in the comment attribute, again)
# @direction: inbound
# @confidence: 60
10.0.0.0/8

 

How to

  1. Download the script from this webpage and save it in a file called minemeld-sync.py. If you have wget installed you can use:
    wget -O minemeld-sync.py https://gist.githubusercontent.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785/raw
  2. Use the following command line to upload the IPv4 indicators stored in the file indicators.lst to the MineMeld instance with URL https://192.168.1.1, using the username admin and password minemeld:
    python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
  3. (OPTIONAL) By default old indicators are not removed from the Miner. If you prefer a full sync of the local list with the remote miner, add the options --update and --delete to the command line:

     

    python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --delete --update --share-level red wlWhiteListIPv4 indicators.lst

 

Remote certificate verification

By default remote MineMeld certificate is verified using certifi package (if installed), or using the CA bundle file or CA certs directory specified via the --ca-path option:


python minemeld-sync.py -m https://192.168.1.1 --ca-path /etc/ssl/certs -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst

 

To disable remote certificate verification use the option -k:


python minemeld-sync.py -m https://192.168.1.1 -k -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst

 

Rate this article:
Comments
L1 Bithead

Thank you for this Imori!

L1 Bithead

 Comment field in the script would be much appreciated Luigi.  I like to mark why I am blacklisting an IP and have record of what it was weeks later when I am looking at the IP Addresses as a reminder. Possible?

 

I am doing 632 comments manually right now in the GUI after running the script to add all those IP addresses to my Miner. 

L7 Applicator

Hi @Kyle_Buffington,

absolutely, that's something I wanted to add. About the syntax, would something like this work ?

# comment for indicator1
indicator1
# comment for indicator2
indicator2
...

Example

# Google Public DNS 1
8.8.8.8
# Google Public DNS 2
8.8.4.4
L1 Bithead

Hello,

Following up on the previous comment, I think it will be useful to have comment, confidence and share level as part of the file. I tweaked your script to take as input a csv file with those fields. Code below.

 

 import csv

def _merge_lists(file_paths):
    result = {}
    for l in file_paths:
        with open(l) as csvfile:
            ioclist = csv.DictReader(csvfile)
            for row in ioclist:
                value['share_level'] = row['share_level']
                value['confidence'] = row['Confidence']
                value['comment'] = row['Description'] +'. '+ row['Source'] +'. ' + row['Tags']
                result[row['Value']] = value
    return result

L1 Bithead

The challenge I have is that I need to import about 20k IOC. I get an HTTP error 413 when I get to about 3700. Any idea how I could solve the issue?

 

L1 Bithead

the script does not seems to be working  when i use python3.7, any idea on what's need to be changed in the script to get it working on 3.7?

 

L0 Member

Hello Imori,

I get the following errors when running your script. Do you know what cause it ?  I jus upgrade to python 2.7.16

 

WARNING:__main__:MineMeld cert verification disabled
Traceback (most recent call last):
File "minemeld-sync.py", line 489, in <module>
main()
File "minemeld-sync.py", line 441, in main
if not MM.check(miner=args.miner, type_=args.type):
File "minemeld-sync.py", line 186, in check
content = self._call_api('/status/minemeld')
File "minemeld-sync.py", line 178, in _call_api
context=self.context
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: UNAUTHORIZED

L0 Member

Hello Imori,

When I use python3.4.3, I get the following error.

 

hguo1@MineMeld:~$ python3 minemeld-sync.py -m https://10.181.44.13 -k -u lo_hguo1 -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
File "minemeld-sync.py", line 209
except urllib2.HTTPError, e:

L0 Member

Awesome, thanks Lmori!

Python2.7 env worked as expected 🙂

L0 Member

Hi,

 

I forked the code and ported to python 3.

 

https://gist.github.com/HugeekMcGill

 

Regards,

 

Hugo

L0 Member

I saw a weird behaviour when using the script while building a list of custom IoCs.

 

During my testing, I needed to flush the previous IoCs from the minemeld whitelist. Using the --delete and an empty list of IoCs, I was not able to flush the whitelist on minemeld but it looks like the scipt worked (Logs shows that entries were deleted ).

 

After some in-dept testing, I figured that we can flush a whitelist of the IoC from minemeld by adding an none empty list. Is that behavior "as design" to protect executing with an empty list and having the risk of flushing the entire whitelist ?

 

 

the I was not ablefor It happened when testing whitelisting, I needed to clea

When I want to clear the whitelist,I'm trying to

L0 Member

I'm trying to upload a list of IPs and keep getting the following error.  I've tried on a system and directly on the Minemeld server VM.  Appreciate any assistance.

 

Traceback (most recent call last):

  File "minemeld-sync.py", line 492, in <module>

    main()

  File "minemeld-sync.py", line 486, in main

    MM.upload(args.miner, '[{}]'.format(','.join(result.values())))

  File "minemeld-sync.py", line 226, in upload

    method='PUT'

  File "minemeld-sync.py", line 174, in _call_api

    context=self.context

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen

    return opener.open(url, data, timeout)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open

    response = meth(req, response)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response

    'http', request, response, code, msg, hdrs)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 467, in error

    result = self._call_chain(*args)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain

    result = func(*args)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 633, in http_error_302

    new = self.redirect_request(req, fp, code, msg, headers, newurl)

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 594, in redirect_request

    raise HTTPError(req.get_full_url(), code, msg, headers, fp)

urllib2.HTTPError: HTTP Error 301: Moved Permanently

  • 36151 Views
  • 12 comments
  • 3 Likes
Register or Sign-in
Contributors
Labels
Article Dashboard
Version history
Last Updated:
‎10-28-2019 02:46 PM
Updated by:
Retired Member