Manually Install MineMeld on Ubuntu 16.04

lmori · ‎03-12-2019

Note: Palo Alto Networks made an end-of-life announcement about the MineMeld™ application in AutoFocus™ on August 1, 2021. Please read this article to learn about our recommended migration options.

An easy way to use MineMeld is installing the binary packages on an Ubuntu 16.04 instance.

Note: Binary packages are only available for 64 bits architectures.

Supported distributions

Ubuntu Server LTS 16.04 (64 bits)

1. Hardening the instance

First thing you should harden your new instance. MineMeld won't take of this for you. A good tutorial is this one.

2. Update the instance

Update all the packages of the instance before installing.

sudo apt update && sudo apt dist-upgrade -y

3. Setting up iptables

You can use the following commands to configure iptables to allow sessions on ports used by MineMeld. Also these rules drop all IPv6 traffic, if you are running MineMeld in an IPv6 network make sure you change the suggested rules.

sudo apt install -y iptables-persistent
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 13514 -j ACCEPT
sudo iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
sudo iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
sudo iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
sudo iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo bash -c "iptables-save > /etc/iptables/rules.v4"
sudo ip6tables -A INPUT -i lo -j ACCEPT
sudo ip6tables -P INPUT DROP
sudo ip6tables -P FORWARD DROP
sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"

4. Adding the repo GPG key

Add the MineMeld repo GPG key to the APT trusted keyring:

 wget -qO - https://minemeld-updates.panw.io/gpg.key | sudo apt-key add -

Double check the GPG key fingerprint, to make sure it is matching the official MineMeld GPG key (fingerprint should match characters in bold):

apt-key adv --fingerprint DD0DA1F9
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.W74MaAG3pI --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --fingerprint DD0DA1F9
pub 4096R/DD0DA1F9 2016-07-15
 Key fingerprint = E558 CE6E 3968 0F31 8F6C BFAC B401 E02E DD0D A1F9
uid Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com>
[...]

5. Adding the MineMeld APT repo

Add the MineMeld APT repo to the system list and update the apt cache:

sudo add-apt-repository "deb http://minemeld-updates.panw.io/ubuntu xenial-minemeld main"
sudo apt update

6. Installing nginx and redis

MineMeld requires nginx and redis. We have to install them before installing MineMeld package to avoid configuration conflicts:

sudo apt install -y nginx redis-server

7. Installing MineMeld

Install the MineMeld infrastructure package via apt. This will also automatically trigger the download of the latest MineMeld packages.

sudo apt install -o Dpkg::Options::="--force-overwrite" -y minemeld

8. Restart

We should restart the instance to make sure all the configurations are applied and all the services are started in the right order:

sudo shutdown -r now

9. Checking if MineMeld is running

Check if the 3 MineMeld services are up and running:

$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine RUNNING pid 3727, uptime 0:08:50
minemeld-traced RUNNING pid 3728, uptime 0:08:50
minemeld-web RUNNING pid 3729, uptime 0:08:50
minemeld-supervisord-listener RUNNING pid 3730, uptime 0:08:50

10. BAM!

Done! Check the Quick Tour article to get started.

Eshrak · ‎03-12-2019

Hello,

I am getting these errors after installing minemeld. System returned no error during install.

Any thoughts would be highly appriciated.

Thanks,

administrator@ubuntu:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf status
[sudo] password for administrator:
sudo: /opt/minemeld/engine/current/bin/supervisorctl: command not found
administrator@ubuntu:~$ ps -ef | grep mine
adminis+ 937 921 0 14:58 pts/0 00:00:00 grep --color=auto mine
administrator@ubuntu:~$ systemctl start minemeld
Failed to start minemeld.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files
See system logs and 'systemctl status minemeld.service' for details.
administrator@ubuntu:~$ systemctl status minemeld.service
● minemeld.service - Process Monitoring and Control Daemon
Loaded: loaded (/lib/systemd/system/minemeld.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-03-12 14:57:54 PDT; 1min 32s ago
Process: 882 ExecStart=/opt/minemeld/engine/current/bin/supervisord -c /opt/minemeld/supervisor/config/supervisord.conf --pidfile /var/run/minemeld/minemeld.pid (code=exited, status=203/EXEC)
Process: 876 ExecStartPre=/bin/chown -R minemeld:minemeld /var/run/minemeld/ (code=exited, status=0/SUCCESS)
Process: 863 ExecStartPre=/bin/mkdir /var/run/minemeld (code=exited, status=0/SUCCESS)

Mar 12 14:57:54 ubuntu systemd[1]: Starting Process Monitoring and Control Daemon...
Mar 12 14:57:54 ubuntu systemd[1]: minemeld.service: Control process exited, code=exited status=203
Mar 12 14:57:54 ubuntu systemd[1]: Failed to start Process Monitoring and Control Daemon.
Mar 12 14:57:54 ubuntu systemd[1]: minemeld.service: Unit entered failed state.
Mar 12 14:57:54 ubuntu systemd[1]: minemeld.service: Failed with result 'exit-code'.

lmori · ‎03-13-2019

Hi @Eshrak, it seems that the installer wasn't able to download the MM packages. Could you try running the following command? If you still have an error, please open a discussion

sudo /usr/sbin/minemeld-auto-update

Eshrak · ‎03-13-2019

Hi @lmori ,

I have a discussion open with the issue.

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-install-errors-on-ubuntu-server-1...

Please advise.

Thanks!

brian.mcdonald · ‎04-24-2019

is ubuntu 19.04 supported?

getting the following error

PLAY [minemeld playbook] ********************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [infrastructure : debug] ***************************************************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
"msg": "Loading vars for Ubuntu 19.04"
}

TASK [infrastructure : include_vars] ********************************************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}
to retry, use: --limit @/home/brian/minemeld-ansible/local.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=2 changed=0 unreachable=0 failed=1

lmori · ‎04-26-2019

@brian.mcdonald Not tested yet on 19.04

srogatnev · ‎05-10-2019

Awesome step by step instruction.

Pls add to preamble "Supported distribution" - "no apache/apache2/httpd service installed/running"

It will save some time to not soo savvy in linux.

aduncan · ‎06-27-2019

Step 1 Issues:

If you see the following just after you entered “sudo apt update && sudo apt dist-upgrade -y”:

….

Err:2 cdrom://Ubuntu-Server 16.04.6 LTS _Xenial Xerus_ - Release amd64 (20190226) xenial Release

Please use apt-cdrom to make this CD-ROM recognized by APT. apt-get update cannot be used to add new CD-ROMs

….

Do the following:

sudo nano /etc/apt/sources.list

and comment or delte the following line

deb cdrom:[Ubuntu-Server 16.04 LTS _Xenial Xerus_ - Release amd64 (20160420.3)]/ xenial main restricted

(NOTE - where it says 20160420.3 it might be a different number for you, so don’t get hung up on it!)

Step 5 issues:

If you get “sudo: add-apt-repository: command not found”, type the following:

sudo apt install software-properties-common

FYI - I'm running MM on VMware 10.1.5 and using the Ubuntu-16.04.6-server-amd64.iso

Tip

I installed open-ssh before gonig through the steps for my VM to make entering commands easyer - to do this enter the following:

sudo apt-get install openssh-server

LarsAtConsigas · ‎07-06-2019

I installed Minemeld on a fresh Ubuntu 16.04.6 LTS with the instructions above but the minemeld-engine is hanging in STARTING state. minemeld-engine.log shows a couple of directory not found messages for CheckPoints. Any idea how to fix this?

lab-user@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
[sudo] password for lab-user:
minemeld-engine                  STARTING
minemeld-supervisord-listener    RUNNING   pid 1190, uptime 0:00:19
minemeld-traced                  RUNNING   pid 1192, uptime 0:00:19
minemeld-web                     RUNNING   pid 1193, uptime 0:00:19

2019-07-06T18:21:20 (4767)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T18:21:20 (4767)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T18:21:21 (4767)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T18:21:21 (4767)config._load_config_from_dir INFO: Changes in config: [_ConfigChange(nodename='spamhaus_EDROP', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='dshield_blocklist', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='inboundaggregator', nodeclass='minemeld.ft.ipop.AggregateIPv4FT', change=0, detail=None), _ConfigChange(nodename='inboundfeedhc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None), _ConfigChange(nodename='spamhaus_DROP', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='wlWhiteListIPv4', nodeclass='minemeld.ft.local.YamlIPv4FT', change=0, detail=None), _ConfigChange(nodename='inboundfeedlc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None), _ConfigChange(nodename='inboundfeedmc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None)]
2019-07-06T18:21:21 (4767)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T18:21:21 (4767)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[_ConfigChange(nodename='spamhaus_EDROP', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='dshield_blocklist', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='inboundaggregator', nodeclass='minemeld.ft.ipop.AggregateIPv4FT', change=0, detail=None), _ConfigChange(nodename='inboundfeedhc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None), _ConfigChange(nodename='spamhaus_DROP', nodeclass='minemeld.ft.http.HttpFT', change=0, detail=None), _ConfigChange(nodename='wlWhiteListIPv4', nodeclass='minemeld.ft.local.YamlIPv4FT', change=0, detail=None), _ConfigChange(nodename='inboundfeedlc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None), _ConfigChange(nodename='inboundfeedmc', nodeclass='minemeld.ft.redis.RedisSet', change=0, detail=None)])
2019-07-06T18:21:21 (4767)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T18:21:21 (4767)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T18:21:21 (4767)launcher.main INFO: Number of chassis: 1
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 10, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 252, in main
    minemeld.comm.cleanup(config.fabric['class'], config.fabric['config'])
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/__init__.py", line 14, in cleanup
    return ZMQRedis.cleanup(config)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/zmqredis.py", line 855, in cleanup
    tkeys = SR.keys(pattern='mm:topic:*')
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 936, in keys
    return self.execute_command('KEYS', pattern)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2019-07-06T18:21:33 (4953)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T18:21:33 (4953)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T18:21:35 (4953)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T18:21:35 (4953)config._load_config_from_dir INFO: Changes in config: []
2019-07-06T18:21:35 (4953)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T18:21:35 (4953)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[])
2019-07-06T18:21:35 (4953)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T18:21:35 (4953)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T18:21:35 (4953)launcher.main INFO: Number of chassis: 1
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 10, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 252, in main
    minemeld.comm.cleanup(config.fabric['class'], config.fabric['config'])
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/__init__.py", line 14, in cleanup
    return ZMQRedis.cleanup(config)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/zmqredis.py", line 855, in cleanup
    tkeys = SR.keys(pattern='mm:topic:*')
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 936, in keys
    return self.execute_command('KEYS', pattern)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2019-07-06T18:21:36 (4969)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T18:21:36 (4969)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T18:21:38 (4969)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T18:21:38 (4969)config._load_config_from_dir INFO: Changes in config: []
2019-07-06T18:21:38 (4969)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T18:21:38 (4969)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[])
2019-07-06T18:21:38 (4969)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T18:21:38 (4969)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T18:21:38 (4969)launcher.main INFO: Number of chassis: 1
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 10, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 252, in main
    minemeld.comm.cleanup(config.fabric['class'], config.fabric['config'])
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/__init__.py", line 14, in cleanup
    return ZMQRedis.cleanup(config)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/zmqredis.py", line 855, in cleanup
    tkeys = SR.keys(pattern='mm:topic:*')
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 936, in keys
    return self.execute_command('KEYS', pattern)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2019-07-06T18:21:40 (4974)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T18:21:40 (4974)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T18:21:42 (4974)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T18:21:42 (4974)config._load_config_from_dir INFO: Changes in config: []
2019-07-06T18:21:42 (4974)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T18:21:42 (4974)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[])
2019-07-06T18:21:42 (4974)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T18:21:42 (4974)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T18:21:42 (4974)launcher.main INFO: Number of chassis: 1
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 10, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 252, in main
    minemeld.comm.cleanup(config.fabric['class'], config.fabric['config'])
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/__init__.py", line 14, in cleanup
    return ZMQRedis.cleanup(config)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/zmqredis.py", line 855, in cleanup
    tkeys = SR.keys(pattern='mm:topic:*')
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 936, in keys
    return self.execute_command('KEYS', pattern)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2019-07-06T18:21:46 (4979)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T18:21:46 (4979)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T18:21:47 (4979)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T18:21:47 (4979)config._load_config_from_dir INFO: Changes in config: []
2019-07-06T18:21:47 (4979)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T18:21:47 (4979)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[])
2019-07-06T18:21:47 (4979)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T18:21:47 (4979)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T18:21:47 (4979)launcher.main INFO: Number of chassis: 1
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 10, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 252, in main
    minemeld.comm.cleanup(config.fabric['class'], config.fabric['config'])
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/__init__.py", line 14, in cleanup
    return ZMQRedis.cleanup(config)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/comm/zmqredis.py", line 855, in cleanup
    tkeys = SR.keys(pattern='mm:topic:*')
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 936, in keys
    return self.execute_command('KEYS', pattern)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2019-07-06T20:22:28 (1191)launcher.main INFO: Starting mm-run.py version 0.9.62
2019-07-06T20:22:28 (1191)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2019-07-06T20:22:30 (1191)config._load_config_from_dir INFO: Switching to candidate config
2019-07-06T20:22:30 (1191)config._load_config_from_dir INFO: Changes in config: []
2019-07-06T20:22:30 (1191)config._destroy_old_nodes INFO: Destroyed nodes: []
2019-07-06T20:22:30 (1191)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'ZMQRedis'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'ZMQRedis'}}, changes=[])
2019-07-06T20:22:30 (1191)launcher.main INFO: multiprocessing: #cores: 2
2019-07-06T20:22:30 (1191)launcher.main INFO: multiprocessing: max #chassis: 2
2019-07-06T20:22:30 (1191)launcher.main INFO: Number of chassis: 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2019-07-06T20:22:30 (1203)base.read_checkpoint ERROR: spamhaus_EDROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_EDROP.chkp'
2019-07-06T20:22:30 (1203)base.state INFO: spamhaus_EDROP - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2019-07-06T20:22:30 (1203)base.read_checkpoint ERROR: dshield_blocklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'dshield_blocklist.chkp'
2019-07-06T20:22:30 (1203)base.state INFO: dshield_blocklist - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2019-07-06T20:22:30 (1203)redis.read_checkpoint ERROR: inboundfeedlc - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 69, in read_checkpoint
    raise ValueError('{} - last checkpoint not found'.format(self.name))
ValueError: inboundfeedlc - last checkpoint not found
2019-07-06T20:22:30 (1203)base.connect INFO: inboundfeedlc - requesting fabric sub channel for inboundaggregator
2019-07-06T20:22:30 (1203)base.state INFO: inboundfeedlc - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2019-07-06T20:22:30 (1203)redis.read_checkpoint ERROR: inboundfeedhc - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 69, in read_checkpoint
    raise ValueError('{} - last checkpoint not found'.format(self.name))
ValueError: inboundfeedhc - last checkpoint not found
2019-07-06T20:22:30 (1203)base.connect INFO: inboundfeedhc - requesting fabric sub channel for inboundaggregator
2019-07-06T20:22:30 (1203)base.state INFO: inboundfeedhc - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2019-07-06T20:22:30 (1203)base.read_checkpoint ERROR: spamhaus_DROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_DROP.chkp'
2019-07-06T20:22:30 (1203)base.state INFO: spamhaus_DROP - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.local.YamlIPv4FT
2019-07-06T20:22:30 (1203)base.read_checkpoint ERROR: wlWhiteListIPv4 - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'wlWhiteListIPv4.chkp'
2019-07-06T20:22:30 (1203)base.state INFO: wlWhiteListIPv4 - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ipop.AggregateIPv4FT
2019-07-06T20:22:30 (1203)base.read_checkpoint ERROR: inboundaggregator - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'inboundaggregator.chkp'
2019-07-06T20:22:30 (1203)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_DROP
2019-07-06T20:22:30 (1203)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_EDROP
2019-07-06T20:22:30 (1203)base.connect INFO: inboundaggregator - requesting fabric sub channel for dshield_blocklist
2019-07-06T20:22:30 (1203)base.connect INFO: inboundaggregator - requesting fabric sub channel for wlWhiteListIPv4
2019-07-06T20:22:30 (1203)base.state INFO: inboundaggregator - transitioning to state 1
2019-07-06T20:22:30 (1203)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2019-07-06T20:22:30 (1203)redis.read_checkpoint ERROR: inboundfeedmc - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 69, in read_checkpoint
    raise ValueError('{} - last checkpoint not found'.format(self.name))
ValueError: inboundfeedmc - last checkpoint not found
2019-07-06T20:22:30 (1203)base.connect INFO: inboundfeedmc - requesting fabric sub channel for inboundaggregator
2019-07-06T20:22:30 (1203)base.state INFO: inboundfeedmc - transitioning to state 1
2019-07-06T20:22:30 (1191)mgmtbus.init_graph INFO: state: {u'mbus:slave:wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}}
2019-07-06T20:22:30 (1191)mgmtbus.init_graph INFO: changes: []
2019-07-06T20:22:30 (1191)startupplanner._plan_subgraph INFO: state_info: {u'spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}}
2019-07-06T20:22:30 (1191)startupplanner._plan_subgraph INFO: planning for subgraph ['spamhaus_EDROP', 'dshield_blocklist', 'inboundfeedlc', 'inboundfeedhc', 'spamhaus_DROP', 'wlWhiteListIPv4', 'inboundaggregator', 'inboundfeedmc']
2019-07-06T20:22:30 (1191)startupplanner._plan_subgraph INFO: No checkpoints, new graph: reset
2019-07-06T20:22:30 (1191)mgmtbus.init_graph INFO: spamhaus_EDROP <= reset
2019-07-06T20:22:30 (1203)base.state INFO: spamhaus_EDROP - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: spamhaus_EDROP - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: dshield_blocklist <= reset
2019-07-06T20:22:31 (1203)base.state INFO: dshield_blocklist - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: dshield_blocklist - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: inboundfeedlc <= reset
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedlc - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedlc - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: inboundfeedhc <= reset
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedhc - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedhc - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: spamhaus_DROP <= reset
2019-07-06T20:22:31 (1203)base.state INFO: spamhaus_DROP - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: spamhaus_DROP - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: wlWhiteListIPv4 <= reset
2019-07-06T20:22:31 (1203)base.state INFO: wlWhiteListIPv4 - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: inboundaggregator <= reset
2019-07-06T20:22:31 (1203)base.state INFO: inboundaggregator - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: inboundaggregator - transitioning to state 4
2019-07-06T20:22:31 (1191)mgmtbus.init_graph INFO: inboundfeedmc <= reset
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedmc - transitioning to state 3
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedmc - transitioning to state 4
2019-07-06T20:22:31 (1203)chassis.mgmtbus_start INFO: chassis - start received from mgmtbus
2019-07-06T20:22:31 (1203)chassis.start INFO: chassis start called
2019-07-06T20:22:31 (1203)base.state INFO: spamhaus_EDROP - transitioning to state 5
2019-07-06T20:22:31 (1203)base.state INFO: dshield_blocklist - transitioning to state 5
2019-07-06T20:22:31 (1203)base.state INFO: inboundaggregator - transitioning to state 5
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedhc - transitioning to state 5
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1562440951063 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: dshield_blocklist - command: 1562440951064 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._huppable_wait INFO: hup is clear: False
2019-07-06T20:22:31 (1203)base.state INFO: spamhaus_DROP - transitioning to state 5
2019-07-06T20:22:31 (1203)base.state INFO: wlWhiteListIPv4 - transitioning to state 5
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedlc - transitioning to state 5
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1562440951065 poll
2019-07-06T20:22:31 (1203)basepoller._polling_loop INFO: Polling spamhaus_EDROP
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_DROP - command: 1562440951077 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562440951077 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._huppable_wait INFO: hup is clear: False
2019-07-06T20:22:31 (1203)base.state INFO: inboundfeedmc - transitioning to state 5
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_DROP - command: 1562440951078 poll
2019-07-06T20:22:31 (1203)basepoller._polling_loop INFO: Polling spamhaus_DROP
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1562440951065 sudden_death
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1562440951065 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1562440951065 gc
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_DROP - command: 1562440951078 sudden_death
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_DROP - command: 1562440951078 age_out
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:31 (1203)basepoller._actor_loop INFO: spamhaus_DROP - command: 1562440951078 gc
2019-07-06T20:22:31 (1203)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-07-06T20:22:32 (1203)launcher._run_chassis INFO: Nodes initialized
2019-07-06T20:22:33 (1203)basepoller._huppable_wait INFO: hup is clear: False
2019-07-06T20:22:33 (1203)basepoller._huppable_wait INFO: hup is clear: False
2019-07-06T20:22:33 (1203)basepoller._actor_loop INFO: dshield_blocklist - command: 1562440953183 poll
2019-07-06T20:22:33 (1203)basepoller._polling_loop INFO: Polling dshield_blocklist
2019-07-06T20:22:33 (1203)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562440953183 poll
2019-07-06T20:22:33 (1203)basepoller._polling_loop INFO: Polling wlWhiteListIPv4
2019-07-06T20:22:33 (1203)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562440953183 sudden_death
2019-07-06T20:22:33 (1203)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562440953183 age_out
2019-07-06T20:22:33 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:33 (1203)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562440953183 gc
2019-07-06T20:22:33 (1203)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-07-06T20:22:34 (1203)basepoller._actor_loop INFO: dshield_blocklist - command: 1562440953183 sudden_death
2019-07-06T20:22:34 (1203)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-07-06T20:22:34 (1203)basepoller._actor_loop INFO: dshield_blocklist - command: 1562440953183 age_out
2019-07-06T20:22:34 (1203)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-06T20:22:34 (1203)basepoller._actor_loop INFO: dshield_blocklist - command: 1562440953183 gc
2019-07-06T20:22:34 (1203)table._query_by_index INFO: Deleted in scan of _withdrawn: 0

lmori · ‎07-07-2019

@LarsAtConsigas did you reboot the instance after the install as per instructions?

LarsAtConsigas · ‎07-07-2019

@lmori yes I did, but unfortunately it didn't help.

lmori · ‎07-07-2019

@LarsAtConsigas please open a discussion with your logs, and please check that redis is running.

Luigi

matthewpr · ‎07-08-2019

I am trying to install Minemeld on Ubuntu 16.04. I ran through all the above steps, restarted, and ran the below command to check if it is running:

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

I get the error:

unix:///var/run/minemeld/minemeld.sock no such file

When I try to start the service with:

service minemeld start

I get the error:

Failed to start minemeld.service: Unit redis.service not found.

I have installed redis and checked with:

service redis-server status

I get:

● redis-server.service - Advanced key-value store
Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2019-07-07 17:30:19 PDT; 20h ago
Docs: http://redis.io/documentation,
man:redis-server(1)
Main PID: 1278 (redis-server)
Tasks: 3
Memory: 1.8M
CPU: 1min 14.907s
CGroup: /system.slice/redis-server.service
└─1278 /usr/bin/redis-server 127.0.0.1:6379

I have also ran the command:

sudo /usr/sbin/minemeld-auto-update

And get:

2019-07-08 14:01:34,630 INFO:0.9.12 Current status:
2019-07-08 14:01:34,631 INFO:0.9.12 minemeld-engine: current: 0.9.62 latest: 0.9.62
2019-07-08 14:01:34,631 INFO:0.9.12 minemeld-webui: current: 0.9.62 latest: 0.9.62
2019-07-08 14:01:34,631 INFO:0.9.12 minemeld-prototypes: current: 0.9.62 latest: 0.9.62
2019-07-08 14:01:34,678 DEBUG:0.9.12 curl output:
2019-07-08 14:01:34,727 DEBUG:0.9.12 curl output:
2019-07-08 14:01:34,727 DEBUG:0.9.12 gpgv: /usr/bin/gpgv --ignore-time-conflict --keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/minemeld.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg /tmp/mmaupackagesgpgnI2ehE /tmp/mmaupackagesahpjtz
2019-07-08 14:01:34,731 INFO:0.9.12 gpgv output: gpgv: Signature made Thu 27 Jun 2019 04:54:07 AM PDT using RSA key ID 7B630999
gpgv: Good signature from "Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com>"
gpgv:                 aka "[invalid image]"

2019-07-08 14:01:34,734 INFO:0.9.12 No package to deploy, exit

Please advise.

Thanks.

upatino · ‎07-11-2019

Hello all,

I tried to install MineMeld, but at Checking if MineMeld is running I got this error:

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
sudo: /opt/minemeld/engine/current/bin/supervisorctl: command not found

How can I solve this ?

Regards

Tobi · ‎08-07-2019

Unfortunately the hardening tutorial is offline.

lmori · ‎08-09-2019

@JarrodEvoTek please could you open a discussion about your installation problem?

ACMENEWS · ‎09-02-2019

Currently, Ubuntu 19.x is the server version while LTS is at 18.x. Are there instructions for installing MineMeld on current Ubuntu releases?

Thank you.

Reference: https://ubuntu.com/download/server

ACMENEWS · ‎09-03-2019

FWIW, I followed the instructions provided by Lmori on Ubuntu 18.04.3 LTS server, patched with the latest updates as of the time of this post. After the reboot, MineMeld was not listed as a service. I reran the following at which point MineMeld was installed as a service.

sudo apt install -o Dpkg::Options::="--force-overwrite" -y minemeld

I browsed to the server on port 443 and was presented with a login to MineMeld. I login as admin/minemeld but received "Bad Gateway". The log files show the following, which are the same as logs others have posted:

[2019-09-04 01:18:17 +0000] [3499] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
    worker.init_process()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
    super(GeventWorker, self).init_process()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
    __import__(module)
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/flask/main.py", line 3, in <module>
    app = create_app()
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 67, in create_app
    from . import metricsapi  # noqa
  File "/opt/minemeld/engine/0.9.62/local/lib/python2.7/site-packages/minemeld/flask/metricsapi.py", line 19, in <module>
    import rrdtool
ImportError: librrd.so.4: cannot open shared object file: No such file or directory
[2019-09-04 01:18:17 +0000] [3499] [INFO] Worker exiting (pid: 3499)
[2019-09-04 01:18:17 +0000] [3496] [INFO] Shutting down: Master
[2019-09-04 01:18:17 +0000] [3496] [INFO] Reason: Worker failed to boot.

I have found many posts about this issue, yet no solutions. rrdtool itself does exist in the minemeld folder.

Any advice?

Thank you.

Potato-soup · ‎09-12-2019

Is this the preferred method to install Minemeld or should I use the anisible method?

Apart from the linux hardedning and the repository, will the end results be the same?

Cthroop · ‎09-14-2019

FAILS AT

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

sudo: unknown user: minemeld
sudo: unable to inialize policy plugin

Cthroop · ‎09-14-2019

FAILED AT

sudo apt install -o Dpkg::Options::="--force-overwrite" -y minemeld

E: Unalbe to locate package minemeld

lmori · ‎09-17-2019

Hi @Potato-soup,

this mechanism also supports auto updates. I am working on an article on how to use Docker to install MineMeld, that is the favorite mechanism.

@Cthroop, are you using a 64bit Ubuntu instance? Could you open a discussion to check your issues?

Unlock your full community experience!

Manually Install MineMeld on Ubuntu 16.04