MineMeld Docker

SabreAce33 — Tue, 26 Jul 2016 13:10:52 GMT

I started building out a very simple dev (read: unhardened) docker build for MineMeld here: https://github.com/swannysec/MineMeldDocker

Looks like it won't start up correctly inside a container and I think it might be related to the use of UNIX sockets and/or something to do with supervisor. Anyway, feel free to take a stab at it! I have to put it aside for a few weeks, but feel free to submit PRs if you find something.

Re: MineMeld Docker

lmori — Mon, 25 Jul 2016 20:08:43 GMT

Hi SabreAce33,

that's super interesting. Also @nbilal was interested in running MineMeld in a docker. I have forked your repo, will take a look at it.

Thanks !

luigi

Re: MineMeld Docker

lmori — Wed, 21 Sep 2016 05:47:59 GMT

@SabreAce33, @nbilal I have started working on the docker container for MineMeld. Still unstable, do not use it in production.

You can pull it from docker hub using:

docker pull jtschichold/minemeld

Git repo available at https://github.com/jtschichold/minemeld-docker

Re: MineMeld Docker

nbilal — Wed, 26 Oct 2016 16:52:34 GMT

When I first run the container, I have to start up the redis-server manually in order to get to the web UI and login. After successful login, I see "ERROR RETRIEVING X: INTERNAL SERVER ERROR" and "ERROR RETRIEVING X: 500" on every page. minemeld-web.log displays the following:

root@40ff97c925f4:/# tail -n 100 /opt/minemeld/log/minemeld-web.log 
Exception on /traced/query [GET]
Traceback (most recent call last):
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
 response = self.full_dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
 rv = self.dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
 return func(*args, **kwargs)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
 'query': query
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
 self._open_channel()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
 self.comm.start()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
 amqp.connection.Connection(**self.amqp_config)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
 self.transport = self.Transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
 return create_transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
 return TCPTransport(host, connect_timeout)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
 raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:41 +0000] "GET /traced/query?nl=100&ts=1477498421474&uuid=6ee01212-ede5-4287-9cd9-dd17042ad65f HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Loaded user: admin
Loaded user: admin
Exception on /traced/query [GET]
Traceback (most recent call last):
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
 response = self.full_dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
 rv = self.dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
 return func(*args, **kwargs)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
 'query': query
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
 self._open_channel()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
 self.comm.start()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
 amqp.connection.Connection(**self.amqp_config)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
 self.transport = self.Transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
 return create_transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
 return TCPTransport(host, connect_timeout)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
 raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:44 +0000] "GET /traced/query?nl=200&ts=1477498424532&uuid=7522a48c-d14a-4265-87af-40bd8f7ea5fc HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Loaded user: admin
Loaded user: admin
Exception on /traced/query [GET]
Traceback (most recent call last):
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
 response = self.full_dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
 rv = self.handle_user_exception(e)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
 reraise(exc_type, exc_value, tb)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
 rv = self.dispatch_request()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
 return self.view_functions[rule.endpoint](**req.view_args)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
 return func(*args, **kwargs)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
 'query': query
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
 self._open_channel()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
 self.comm.start()
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
 amqp.connection.Connection(**self.amqp_config)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
 self.transport = self.Transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
 return create_transport(host, connect_timeout, ssl)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
 return TCPTransport(host, connect_timeout)
 File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
 raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:47 +0000] "GET /traced/query?nl=200&ts=1477498427729&uuid=8e85bce3-58ee-4de7-bca0-f16ed5a0a5be HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
root@40ff97c925f4:/#

And minemeld-engine.log displays the following:

root@40ff97c925f4:/# tail -n 100 /opt/minemeld/log/minemeld-engine.log 
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Process Process-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
2016-10-26T16:48:46 (14247)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 187, in main
    config['nodes'].keys()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 71, in _start_mgmtbus_master
    mbusmaster.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 323, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
root@40ff97c925f4:/#

It does indeed sound like a socket related error, have no idea what that is or how to fix it, but hopefully somebody more qualified will find this info useful. Maybe something broken in RabbitMQ configuration that needs to be adapted to docker setup, as found here and here?

Re: MineMeld Docker

lmori — Thu, 27 Oct 2016 08:18:01 GMT

Hi @nbilal,

please, could you post your docker logs ? you can grab them using the following command (I am pretty sure you already know this):

docker logs <container>

It seems a problem with RabbitMQ in your environment, could you check:

- if IPv6 is enabled on the docker host ? RabbitMQ release in Ubuntu 14.04 has a bug and you need IPv6 enabled on the docker host even if you are not forwarding it. That's why in the default MineMeld VM IPv6 is enabled but blocked via iptables.

- the /var/log/rabbitmq/startup_err log file ?

Thanks,

luigi

Re: MineMeld Docker

nbilal — Mon, 14 Nov 2016 19:55:11 GMT

Just updated my docker-compose this morning o 1.90rc and thought I'd try "docker-compose up" again just for kicks...and everything's working! Did you make any changes to your container on hub.docker.com @lmori?

nbilal@e6430 ~/Docker/minemeld $ docker inspect minemeld_minemeld_1 | grep Image
        "Image": "sha256:0284c3d01bed0094a4466b501207cedefca100a969b64f7389c5107674a9d9cd",
            "Image": "jtschichold/minemeld",

Keep up the great work!

Re: MineMeld Docker

lmori — Mon, 14 Nov 2016 22:13:50 GMT

Yes, when I released 0.9.26 I have added some improvements to the stability of the docker image and the docker-compose file.

Glad it worked !

topic Re: MineMeld Docker in General Topics

MineMeld Docker

Re: MineMeld Docker

Re: MineMeld Docker

Re: MineMeld Docker

Re: MineMeld Docker

Re: MineMeld Docker

Re: MineMeld Docker