on 06-20-2017 05:18 AM - edited on 10-28-2019 02:46 PM by Retired Member
MineMeld supports Miners where the list of indicators is stored in a local database inside the MineMeld instance. These Miners can be used to define a static list of malicious indicators or a static whitelist (for more details about using a Miner as a whitelist check the article https://live.paloaltonetworks.com/t5/MineMeld-Articles/Creating-whitelists/ta-p/72250).
You can add, delete and change the indicators stored in these Miners directly from the WebUI (the following screenshot shows an example of a whitelist, same applies to malicious indicator lists):
|
stdlib.listDomainGeneric |
List of domain indicators |
| stdlib.listIPv4Generic | List of IPv4 indicators |
| stdlib.listIPv6Generic | List of IPv6 indicators |
| stdlib.listURLGeneric | List of URL indicators |
Using the MineMeld API you can upload indicators to Miners of local indicators. This can be automated using the minemeld-sync.py script: https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785
To use the script Python 2.7.9+ is required. If you are planning to use the script from Linux or Mac OS X, python should be already available in your environment. Otherwise you can download and install python from here https://www.python.org
The list of indicators to be uploaded should be stored in a plain text file, one line per indicator:
8.8.8.8
8.8.4.4
10.0.0.0/8
You can also add comments to each indicator, to be stored in the indicator comment attribute in MineMeld:
# Google public DNS (this will be placed in the comment attribute)
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
8.8.4.4
# Private network (in the comment attribute, again)
10.0.0.0/8
You can also specify custom attributes, with the format @<attribute name>: <attribute value>:
# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.8.8
# Google public DNS (this will be placed in the comment attribute)
# @direction: inbound
8.8.4.4
# Private network (in the comment attribute, again)
# @direction: inbound
# @confidence: 60
10.0.0.0/8
wget -O minemeld-sync.py https://gist.githubusercontent.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785/raw
python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --delete --update --share-level red wlWhiteListIPv4 indicators.lst
By default remote MineMeld certificate is verified using certifi package (if installed), or using the CA bundle file or CA certs directory specified via the --ca-path option:
python minemeld-sync.py -m https://192.168.1.1 --ca-path /etc/ssl/certs -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst
To disable remote certificate verification use the option -k:
python minemeld-sync.py -m https://192.168.1.1 -k -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst
Comment field in the script would be much appreciated Luigi. I like to mark why I am blacklisting an IP and have record of what it was weeks later when I am looking at the IP Addresses as a reminder. Possible?
I am doing 632 comments manually right now in the GUI after running the script to add all those IP addresses to my Miner.
Hi @Kyle_Buffington,
absolutely, that's something I wanted to add. About the syntax, would something like this work ?
# comment for indicator1 indicator1 # comment for indicator2 indicator2 ...
Example
# Google Public DNS 1 8.8.8.8 # Google Public DNS 2 8.8.4.4
Hello,
Following up on the previous comment, I think it will be useful to have comment, confidence and share level as part of the file. I tweaked your script to take as input a csv file with those fields. Code below.
import csv
def _merge_lists(file_paths):
result = {}
for l in file_paths:
with open(l) as csvfile:
ioclist = csv.DictReader(csvfile)
for row in ioclist:
value['share_level'] = row['share_level']
value['confidence'] = row['Confidence']
value['comment'] = row['Description'] +'. '+ row['Source'] +'. ' + row['Tags']
result[row['Value']] = value
return result
Hello Imori,
I get the following errors when running your script. Do you know what cause it ? I jus upgrade to python 2.7.16
WARNING:__main__:MineMeld cert verification disabled
Traceback (most recent call last):
File "minemeld-sync.py", line 489, in <module>
main()
File "minemeld-sync.py", line 441, in main
if not MM.check(miner=args.miner, type_=args.type):
File "minemeld-sync.py", line 186, in check
content = self._call_api('/status/minemeld')
File "minemeld-sync.py", line 178, in _call_api
context=self.context
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: UNAUTHORIZED
Hello Imori,
When I use python3.4.3, I get the following error.
hguo1@MineMeld:~$ python3 minemeld-sync.py -m https://10.181.44.13 -k -u lo_hguo1 -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst
File "minemeld-sync.py", line 209
except urllib2.HTTPError, e:
I saw a weird behaviour when using the script while building a list of custom IoCs.
During my testing, I needed to flush the previous IoCs from the minemeld whitelist. Using the --delete and an empty list of IoCs, I was not able to flush the whitelist on minemeld but it looks like the scipt worked (Logs shows that entries were deleted ).
After some in-dept testing, I figured that we can flush a whitelist of the IoC from minemeld by adding an none empty list. Is that behavior "as design" to protect executing with an empty list and having the risk of flushing the entire whitelist ?
the I was not ablefor It happened when testing whitelisting, I needed to clea
When I want to clear the whitelist,I'm trying to
I'm trying to upload a list of IPs and keep getting the following error. I've tried on a system and directly on the Minemeld server VM. Appreciate any assistance.
Traceback (most recent call last):
File "minemeld-sync.py", line 492, in <module>
main()
File "minemeld-sync.py", line 486, in main
MM.upload(args.miner, '[{}]'.format(','.join(result.values())))
File "minemeld-sync.py", line 226, in upload
method='PUT'
File "minemeld-sync.py", line 174, in _call_api
context=self.context
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 467, in error
result = self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 633, in http_error_302
new = self.redirect_request(req, fp, code, msg, headers, newurl)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 594, in redirect_request
raise HTTPError(req.get_full_url(), code, msg, headers, fp)
urllib2.HTTPError: HTTP Error 301: Moved Permanently
