MineMeld Docker

L2 Linker

MineMeld Docker

I started building out a very simple dev (read: unhardened) docker build for MineMeld here: https://github.com/swannysec/MineMeldDocker

 

Looks like it won't start up correctly inside a container and I think it might be related to the use of UNIX sockets and/or something to do with supervisor.  Anyway, feel free to take a stab at it!  I have to put it aside for a few weeks, but feel free to submit PRs if you find something.

L7 Applicator

Re: MineMeld Docker

Hi SabreAce33,

that's super interesting. Also @nbilal was interested in running MineMeld in a docker. I have forked your repo, will take a look at it.

 

Thanks !

luigi

L7 Applicator

Re: MineMeld Docker

@SabreAce33@nbilal I have started working on the docker container for MineMeld. Still unstable, do not use it in production.

 

You can pull it from docker hub using:

docker pull jtschichold/minemeld

Git repo available at https://github.com/jtschichold/minemeld-docker

L3 Networker

Re: MineMeld Docker

 

 When I first run the container, I have to start up the redis-server manually in order to get to the web UI and login. After successful login, I see "ERROR RETRIEVING X: INTERNAL SERVER ERROR" and "ERROR RETRIEVING X: 500" on every page. minemeld-web.log displays the following:

root@40ff97c925f4:/# tail -n 100 /opt/minemeld/log/minemeld-web.log 
Exception on /traced/query [GET]
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
'query': query
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
self._open_channel()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
self.comm.start()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:41 +0000] "GET /traced/query?nl=100&ts=1477498421474&uuid=6ee01212-ede5-4287-9cd9-dd17042ad65f HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Loaded user: admin
Loaded user: admin
Exception on /traced/query [GET]
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
'query': query
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
self._open_channel()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
self.comm.start()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:44 +0000] "GET /traced/query?nl=200&ts=1477498424532&uuid=7522a48c-d14a-4265-87af-40bd8f7ea5fc HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Loaded user: admin
Loaded user: admin
Exception on /traced/query [GET]
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/tracedapi.py", line 73, in traced_query
'query': query
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 122, in send_cmd
self._open_channel()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 119, in _open_channel
self.comm.start()
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
error: [Errno 111] Connection refused
127.0.0.1 - - [26/Oct/2016:16:13:47 +0000] "GET /traced/query?nl=200&ts=1477498427729&uuid=8e85bce3-58ee-4de7-bca0-f16ed5a0a5be HTTP/1.0" 500 291 "https://localhost:8443/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
root@40ff97c925f4:/#

And minemeld-engine.log displays the following:

root@40ff97c925f4:/# tail -n 100 /opt/minemeld/log/minemeld-engine.log 
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Process Process-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
2016-10-26T16:48:46 (14247)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/chassis.py", line 100, in configure
    self.mgmtbus.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 373, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 187, in main
    config['nodes'].keys()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 71, in _start_mgmtbus_master
    mbusmaster.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 323, in start
    self.comm.start()
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 545, in start
    amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/0.9.24/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
root@40ff97c925f4:/# 

 

It does indeed sound like a socket related error, have no idea what that is or how to fix it, but hopefully somebody more qualified will find this info useful. Maybe something broken in RabbitMQ configuration that needs to be adapted to docker setup, as found here and here?

 

Tags (3)
L7 Applicator

Re: MineMeld Docker

Hi @nbilal,

please, could you post your docker logs ? you can grab them using the following command (I am pretty sure you already know this):

docker logs <container>

It seems a problem with RabbitMQ in your environment, could you check:

- if IPv6 is enabled on the docker host ? RabbitMQ release in Ubuntu 14.04 has a bug and you need IPv6 enabled on the docker host even if you are not forwarding it. That's why in the default MineMeld VM IPv6 is enabled but blocked via iptables.

- the /var/log/rabbitmq/startup_err log file ?

 

Thanks,

luigi

L3 Networker

Re: MineMeld Docker

Just updated my docker-compose this morning o 1.90rc and thought I'd try "docker-compose up" again just for kicks...and everything's working! Did you make any changes to your container on hub.docker.com @lmori?

 

nbilal@e6430 ~/Docker/minemeld $ docker inspect minemeld_minemeld_1 | grep Image
        "Image": "sha256:0284c3d01bed0094a4466b501207cedefca100a969b64f7389c5107674a9d9cd",
            "Image": "jtschichold/minemeld",

 

Keep up the great work!

Highlighted
L7 Applicator

Re: MineMeld Docker

Yes, when I released 0.9.26 I have added some improvements to the stability of the docker image and the docker-compose file.

 

Glad it worked !

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!