Running MineMeld using Docker

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Printer Friendly Page

An easy and powerful way of installing MineMeld is using MineMeld docker image. A docker-based installation of MineMeld can run on any Linux distribution supported by Docker and it is extremely easy to upgrade and maintain.

 

Overview

The procedure to use MineMeld is pretty simple:

The rest of the article will guide you thru installing Docker CE on RHEL 7 and run MineMeld on top of it. Only the first part, the one related to installing Docker on RHEL, is RHEL specific. The second part, the one related to MineMeld itself, is distribution independent.

 

Install Docker on RHEL7

  1. Update your RHEL
    sudo yum update -y
  2. Remove old docker version
    sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
  3. Install container-selinux package from CentOS repo
    sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
  4. Install additional packages
    sudo yum install -y yum-utils   device-mapper-persistent-data   lvm2 container-selinux
  5. Add official Docker CE repo
    sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  6. Rebuild package cache
    sudo yum update -y && sudo yum makecache fast
  7. Install official Docker CE packages
    sudo yum install docker-ce docker-ce-cli containerd.io
  8. Start Docker engine
    sudo systemctl start docker
  9. Make sure Docker is working
    sudo docker run hello-world

 

Install & Run MineMeld

  1. Pull the latest official image
    sudo docker pull paloaltonetworks/minemeld
  2. Create named volumes for data and logs
    sudo docker volume create minemeld-logs
    sudo docker volume create minemeld-local
  3. Start the container
    sudo docker run -dit --name minemeld --restart unless-stopped --tmpfs /run -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log  -p 443:443 -p 80:80 paloaltonetworks/minemeld
  4. After ~30 seconds you should see minemeld running in the container logs
    [ec2-user@minemeld ~]$ sudo docker logs minemeld
    *** Running /etc/rc.local...
    *** Booting runit daemon...
    *** Runit started as PID 7
    minemeld: checking if dependencies are running...
    run: redis: (pid 18) 0s
    run: collectd: (pid 19) 0s
    Copying constraints
    Starting redis-server...
    Regenarating CA bundle
    Sep 18 14:07:31 d0b5d1fbc102 syslog-ng[20]: syslog-ng starting up; version='3.5.6'
    2019-09-18T14:07:31 (35)cacert_merge.main INFO: config: {'cafile': ['/opt/minemeld/local/certs/site/'], 'dst': '/opt/minemeld/local/certs/bundle.crt', 'config': '/opt/minemeld/local/certs/cacert-merge-config.yml', 'no_merge_certifi': False}
    (integer) 0
    Starting minemeld...
    /opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
      return pkg_resources.EntryPoint.parse("x="+spec).load(False)
    2019-09-18 14:07:32,153 CRIT Set uid to user 106
    2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing
    2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing
    2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing
    2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing
    2019-09-18 14:07:32,164 INFO RPC interface 'supervisor' initialized
    2019-09-18 14:07:32,164 CRIT Server 'unix_http_server' running without any HTTP authentication checking
    2019-09-18 14:07:32,164 INFO supervisord started with pid 21
    2019-09-18 14:07:33,167 INFO spawned: 'minemeld-supervisord-listener' with pid 60
    2019-09-18 14:07:33,168 INFO spawned: 'minemeld-engine' with pid 61
    2019-09-18 14:07:33,170 INFO spawned: 'minemeld-traced' with pid 62
    2019-09-18 14:07:33,172 INFO spawned: 'minemeld-web' with pid 63
    2019-09-18 14:07:34,322 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    2019-09-18 14:07:34,322 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    2019-09-18 14:07:34,322 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    2019-09-18 14:08:03,191 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
  5. Connect to the Web interface (https://<address>/) and use username admin and password minemeld to login

 

Check MineMeld Logs

You can check MineMeld engine and MineMeld audit logs from outside the container. Inspect the minemeld-logs volume to grab the directory used by the Docker engine to store volume files. Your logs will be there.

 

[ec2-user@minemeld ~]$ sudo docker inspect minemeld-logs
[
    {
        "CreatedAt": "2019-09-18T10:07:32-04:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/minemeld-logs/_data",
        "Name": "minemeld-logs",
        "Options": {},
        "Scope": "local"
    }
]
[ec2-user@minemeld ~]$ sudo ls /var/lib/docker/volumes/minemeld-logs/_data
minemeld-engine-stderr---supervisor-PV9ZPJ.log		      minemeld-supervisord-listener.log		      minemeld-web-stderr---supervisor-cXauy4.log
minemeld-engine.log					      minemeld-traced-stderr---supervisor-4V3T4E.log  minemeld-web.log
minemeld-supervisord-listener-stdout---supervisor-T3Wfwi.log  minemeld-traced.log			      supervisord.log

 

Backup MineMeld

The config directory in the minemeld-local volume contains all the file needed to restore the MineMeld installation. You can backup the files by inspecting the volume and perform a backup of the config directory.

[ec2-user@minemeld ~]$ sudo docker inspect minemeld-local
[
    {
        "CreatedAt": "2019-09-18T10:03:12-04:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/minemeld-local/_data",
        "Name": "minemeld-local",
        "Options": {},
        "Scope": "local"
    }
]
[ec2-user@minemeld ~]$ sudo ls /var/lib/docker/volumes/minemeld-local/_data
certs  config  data  library  prototypes  redis  supervisor  trace

 

Upgrade MineMeld

To upgrade to the latest and greatest release:

  1. Pull the latest release
    sudo docker pull paloaltonetworks/minemeld
  2. Stop and remove the current container
    sudo docker stop minemeld
    sudo docker rm minemeld
  3. Start the container
    sudo docker run -dit --name minemeld --restart unless-stopped --tmpfs /run -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log  -p 443:443 -p 80:80 paloaltonetworks/minemeld

 

Change MineMeld UI Certificate

The default certificate used by MineMeld is a self-signed certificate. You should change it when running in production. To do it just map your certificate and private key on the default certificate:

sudo docker run -dit \
--name minemeld \
--restart unless-stopped \
--tmpfs /run \
-v minemeld-local:/opt/minemeld/local \
-v minemeld-logs:/opt/minemeld/log \
-v /var/lib/minemeld/real-cert.crt:/etc/nginx/minemeld.cer:ro \
-v /var/lib/minemeld/real-cert.pem:/etc/nginx/minemeld.pem:ro \
-p 443:443 -p 80:80 \
paloaltonetworks/minemeld

 

Tags (5)
Comments

Thanks @lmori this looks great I may give it a go.   I am at that point where I'm about to build our Minemeld solution.   I have successfully created a 16.04 Ubuntu build using anisible in our Dev environment.  All other distro are listed as "Still experimental" on the minemeld site. 

How stable is the Docker install?   I'd be interested in what you would recommend.

 

 

 

@Potato-soup It is stable, it is being used in production.

I can confirm this will work on Ubuntu as well.  Get docker running on Ubuntu using the instructions below. Then follow the instructions from Install & Run MineMeld.

 

https://docs.docker.com/install/linux/docker-ce/ubuntu/

 

Great article @lmori 

Thank you to Ksampson and Lmori.  Following the provided instructions I was able to install MimeMeld within Ubuntu 18.04.1 LTS and have it properly function.   I have been attempting to replace my Ubuntu 14.x MineMeld install and these were the only instructions which actually worked.

It doesn't look like port 13514 is being exposed for the syslog miner. Does the docker container support the syslog miner in this article?

Hello can anyone help?

I am running Unbuntu 18.04.3 LTS and have Minemeld installed and running in docker, however I can't not get my new certificate into the docker and replace the the self-signed certificates for Minemeld.  I have tried SCP/WinSCP/SFTP I just can't seem to get my new certificates in the correct location. #nearlythere! Can anyone point me in the right direction I am a total novice with Docker please bear with!

Hello all,

 

So I am able to get the cert there now as above ^

 

However I have an issue now that I can't restart the NGINX service

 

root@4416471727a8:/etc/nginx# sudo /etc/init.d/nginx restart
* Restarting nginx nginx [fail]

 

This happens even if I do a fresh rebuild of the docker.  Has anyone else had this issue? I can't see any issues?

 

Any help or ideas would be great

I have a strange issue with this. I run throught the setup guide and it works great, minemeld docker instance starts-up ok, I login, change admin password, add another user, logout and back in again without issue, however, as soon as I reboot the VM (it on ESXi) when I go to login I get "Error checking credentials: Timeout" - any help appreciated on getting this fixed is appreciated.

Update, if I login as the new user I created it works great.

 

So, to re-cap.

 

  1. Stand-up new docker minemeld and starts no issue
  2. Login as default admin user - no issues
  3. Change admin password and add another user
  4. Logout, login again as admin user the new password - works great
  5. Reboot VM
  6. Login as admin user with new password -  "Error checking credentials: Timeout"
  7. Login with the newly created user works without issue

I wish this was well documented for macOS users running MineMeld on Docker. "tmpfs" is Linux only so in order to use persistent memory, I created RAM-only volumes and symlink'd each to the respective minemeld volumes; and then modified the syntax to start the container without the tmpfs argument. Would be great if there was proper documentation for this; spent a lot of productive hours just to get this working. Thanks though for all the work gone into MineMeld.

per @jdanjuma's note, tmps wont work on mac, follow steps 1 & 2 and then use the following:

 

sudo docker run -dit --name minemeld --restart unless-stopped -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log -p 443:443 -p 80:80 paloaltonetworks/minemeld

 

launch your browser to localhost, default creds & you're in.

@John_Merry , we're having a bit of trouble with the certificate commands. Can you elaborate on your procedure for uploading and replacing the default certificates?

After getting the docker image running I am seeing an error with the default miners.  I'm not sure how to troubleshoot it as the container doesn't seem to have any network tools to verify access.

 

I've reviewed the resolv.conf file and it has the correct dns servers.  

I've also added the docker host to the an ssl bypass to ensure it isn't getting an ssl certificate verification problem when accessing external sources.

 

minemeld_error.png

 

Any ideas?

From the minemeld server CLI are you able to wget the list as a way to validate your outbound security group isn't blocking?

mdensley_0-1589838326954.png

 

@mdensley 

I am able to successfully perform the wget from the docker host but not from the cli of the container.  It seems to be a name lookup issue:

 

wget https://www.dshield.org/block.txt
--2020-05-19 11:18:54-- https://www.dshield.org/block.txt
Resolving www.dshield.org (www.dshield.org)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘www.dshield.org’

 

I've checked /etc/resolv.conf which has the same dns servers as the docker host.

The issue was the host firewall running on a standard Centos 8 install and blocking outbound connections on the docker interface. 

 

I used the busybox image to troubleshoot connectivity until I was satisfied it was working correctly.  Hope this helps someone else.

@lmori Thank you for putting this together.  It made getting Minemeld running very simple and painless, right up to the point of replacing the certificates.  Would you mind expanding a bit on how to replace the certificate so that the outputs can be used as EDLs?  Thank you in advance.

Hello,


İ cannot run syslog miner, dasboard says it is on, after sending syslog to 13514 nothing happens. anyone knows how to run syslog miner?

*l forwarded port 13514 to container.

 

Thank you.

Do you know how to re install minemeld using docker.

I am facing some Engine Fatal Error, and I want to reset everything and install it again from beginning.

For that I need to remove current details and then have to follow above mention steps.

 

So if you could help me with that.

 

So, I'm using ports 80 and 443 on my docker box, so I re-mapped those ports on the host side to 8080 and 8443, as one does.  Imagine my surprise when I looked at the feed URLs and they had the correct re-mapped ports!  Great job guys!

excuse a docker noob....but is there anyway to patch some of the vulnerabilities in this container like the linux host?

Looks like we have 17 vulns listed with potential patches?

Also wondering how work is going migrating to a python3 stack? Love the product....but moved to docker due to ansible build having dependancy hell breaks relating to the python. I imagine that fixes probably won't be too available until a migration to Python3 happens.

In meantime i have been checking and upgrading docker image as per instructions above.

Screengrab of the Vulnerabilities listed:

Paul_Stinson_0-1600065841882.png

 

Hi @Paul_Stinson,

we are working on the Python3 release. In the meantime you can download the latest docker image with all the Ubuntu security updates applied (we just pushed it). You can also build the image directly from the Dockerfile (https://github.com/PaloAltoNetworks/minemeld-docker), the script will automatically apply all the security updates.

@lmori this appears to have cleared up the vulnerabilities being reported. Awesome.

I had just tried the pull of latest build just before i posted so amazing timing on getting update out after my post! universe in sync, check!

Am looking forward to python 3 upgrade for Minemeld and any future enhancements planned. Thanks for all the good work the team put into the product!

Version history
Revision #:
4 of 4
Last update:
‎09-26-2019 10:29 AM
Updated by:
 
Contributors