Ubuntu 18.04 install errors

[bsmith2]

Hi There, 

Running into issues trying to install on Ubuntu 18.04

During the install process at the end I get this message:

TASK [minemeld : requirements] ******************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": "/opt/minemeld/engine/current/bin/pip2 freeze", "msg": "\n:stderr: Traceback (most recent call last):\n File \"/opt/minemeld/engine/current/bin/pip2\", line 6, in <module>\n from pip._internal import main\n File \"/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pip/_internal/__init__.py\", line 5, in <module>\n import logging\n File \"/usr/lib/python2.7/logging/__init__.py\", line 26, in <module>\n import sys, os, time, cStringIO, traceback, warnings, weakref, collections\n File \"/usr/lib/python2.7/weakref.py\", line 14, in <module>\n from _weakref import (\nImportError: cannot import name _remove_dead_weakref\n"}
to retry, use: --limit @/home/timpo/minemeld-ansible/local.retry

And this is the output of journalctl -xe:

-- Unit minemeld.service has begun starting up.
Apr 10 12:40:43 ise-miner mkdir[16946]: /bin/mkdir: cannot create directory ‘/var/run/minemeld’: File exists
Apr 10 12:40:44 ise-miner supervisord[16948]: Traceback (most recent call last):
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/current/bin/supervisord", line 6, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.supervisord import main
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisord.py", line 41, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.options import ServerOptions
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: import pkg_resources
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: __import__('pkg_resources.extern.packaging.requirements')
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
Apr 10 12:40:44 ise-miner supervisord[16948]: "distribution.".format(**locals())
Apr 10 12:40:44 ise-miner supervisord[16948]: ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the p
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Control process exited, code=exited status=1
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Failed with result 'exit-code'.
Apr 10 12:40:44 ise-miner sudo[16924]: pam_unix(sudo:session): session closed for user root
Apr 10 12:40:44 ise-miner systemd[1]: Failed to start Process Monitoring and Control Daemon.

And here's the output of sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status:

Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/supervisorctl", line 6, in <module>
from supervisor.supervisorctl import main
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisorctl.py", line 36, in <module>
from supervisor.options import ClientOptions
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
import pkg_resources
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
__import__('pkg_resources.extern.packaging.requirements')
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
"distribution.".format(**locals())
ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the packager of your distribution.

Any ideas?

[bsmith2]

I was able to work past installation errors using a fresh install of Ubuntu, however now there's another issue.

[2019-04-10 15:48:52 +0000] [7647] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
worker.init_process()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
super(GeventWorker, self).init_process()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module>
app = create_app()
File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 67, in create_app
from . import metricsapi # noqa
File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 19, in <module>
import rrdtool
ImportError: librrd.so.4: cannot open shared object file: No such file or directory
[2019-04-10 15:48:52 +0000] [7647] [INFO] Worker exiting (pid: 7647)
[2019-04-10 15:48:52 +0000] [7644] [INFO] Shutting down: Master
[2019-04-10 15:48:52 +0000] [7644] [INFO] Reason: Worker failed to boot.

[lmori]

Hi @bsmith2,

which version of Ubuntu 18.04 are you using? Did you follow the instructions before runnig the playbook?

https://github.com/PaloAltoNetworks/minemeld-ansible#howto-on-ubuntu-1804

Luigi

[bsmith2]

Hi Luigi,

Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-47-generic x86_64)

I followed those steps in order, and it seemed to install correctly.

timpo@ise-meld-18:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

[sudo] password for timpo:

minemeld-engine                  RUNNING   pid 8924, uptime 5:13:04

minemeld-supervisord-listener    RUNNING   pid 7609, uptime 18:02:19

minemeld-traced                  FATAL     Exited too quickly (process log may have details)

minemeld-web                     FATAL     Exited too quickly (process log may have details)

Previously minemeld-traced was running but minemeld-web is always fatal.

Thanks!

[lmori]

@bsmith2 could you reboot the instance? if it doesn't work, please check the /opt/minemeld/log/minemeld-web.log file for errors

Thanks,

Luigi

[bsmith2]

Hi Luigi, 

Rebooted the instance and nothing starts, different behavior from previously. Here is the repeating sequence from the weblog:

[2019-04-11 10:02:09 +0000] [648] [INFO] Worker exiting (pid: 648)

[2019-04-11 10:02:09 +0000] [645] [INFO] Shutting down: Master

[2019-04-11 10:02:09 +0000] [645] [INFO] Reason: Worker failed to boot.

[2019-04-11 10:02:13 +0000] [652] [INFO] Starting gunicorn 19.5.0

[2019-04-11 10:02:13 +0000] [652] [INFO] Listening at: http://127.0.0.1:5000 (652)

[2019-04-11 10:02:13 +0000] [652] [INFO] Using worker: gevent

[2019-04-11 10:02:13 +0000] [655] [INFO] Booting worker with pid: 655

[2019-04-11 10:02:13 +0000] [655] [ERROR] Exception in worker process

Traceback (most recent call last):

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker

    worker.init_process()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process

    super(GeventWorker, self).init_process()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process

    self.load_wsgi()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi

    self.wsgi = self.app.wsgi()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi

    self.callable = self.load()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load

    return self.load_wsgiapp()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp

    return util.import_app(self.app_uri)

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app

    __import__(module)

ImportError: No module named minemeld.flask.main

[2019-04-11 10:02:13 +0000] [655] [INFO] Worker exiting (pid: 655)

[2019-04-11 10:02:13 +0000] [652] [INFO] Shutting down: Master

[2019-04-11 10:02:13 +0000] [652] [INFO] Reason: Worker failed to boot.

Thanks!

[bsmith2]

Update - I re-ran the playbook with the version=master switch and I'm getting the same behavior as yesterday, all components start with the exception of web and the weblog now reflects:

[2019-04-11 10:32:24 +0000] [4650] [INFO] Starting gunicorn 19.5.0

[2019-04-11 10:32:24 +0000] [4650] [INFO] Listening at: http://127.0.0.1:5000 (4650)

[2019-04-11 10:32:24 +0000] [4650] [INFO] Using worker: gevent

[2019-04-11 10:32:24 +0000] [4653] [INFO] Booting worker with pid: 4653

[2019-04-11 10:32:25 EDT] [4653] [INFO] MMLogger started

/opt/minemeld/engine/core/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead.

  import flask.ext.login

[2019-04-11 10:32:25 EDT] [4653] [INFO] Config loaded: {'MINEMELD_LOCAL_LIBRARY_PATH': '/opt/minemeld/local/library', 'USERS_DB': 'wsgi.htpasswd', 'MINEMELD_LOCAL_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes', 'SUPERVISOR_URL': 'unix:///var/run/minemeld/minemeld.sock', 'MINEMELD_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes:/opt/minemeld/prototypes/current', 'MINEMELD_PIP_PATH': '/opt/minemeld/engine/current/bin/pip', 'MINEMELD_TRACED_PURGE_PATH': '/opt/minemeld/engine/current/bin/mm-traced-purge', 'API_AUTH_ENABLED': True, 'MINEMELD_LOG_DIRECTORY_PATH': '/opt/minemeld/log', 'MINEMELD_7Z_PATH': '/usr/bin/7z', 'MINEMELD_RESTORE_PATH': '/opt/minemeld/engine/current/bin/mm-restore', 'MINEMELD_LOCAL_PATH': '/opt/minemeld/local', 'MINEMELD_GIT_PATH': '/usr/bin/git', 'SNS_ENABLED': True}

[2019-04-11 10:32:25 EDT] [4653] [INFO] USERS_DB loaded from /opt/minemeld/local/config/api/wsgi.htpasswd

[2019-04-11 10:32:25 EDT] [4653] [INFO] FEEDS_USERS_DB loaded from /opt/minemeld/local/config/api/feeds.htpasswd

[2019-04-11 10:32:25 EDT] [4653] [DEBUG] Instance uuid = ea51fd66d39349079450431b8c259750

[2019-04-11 10:32:25 +0000] [4653] [ERROR] Exception in worker process

Traceback (most recent call last):

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker

    worker.init_process()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process

    super(GeventWorker, self).init_process()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process

    self.load_wsgi()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi

    self.wsgi = self.app.wsgi()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi

    self.callable = self.load()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load

    return self.load_wsgiapp()

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp

    return util.import_app(self.app_uri)

  File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app

    __import__(module)

  File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module>

    app = create_app()

  File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 67, in create_app

    from . import metricsapi  # noqa

  File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 19, in <module>

    import rrdtool

ImportError: librrd.so.4: cannot open shared object file: No such file or directory

[2019-04-11 10:32:25 +0000] [4653] [INFO] Worker exiting (pid: 4653)

[2019-04-11 10:32:25 +0000] [4650] [INFO] Shutting down: Master

[2019-04-11 10:32:25 +0000] [4650] [INFO] Reason: Worker failed to boot.

[bsmith2]

Another update, I managed to get it running by installing the rrdtool dev package and then sym linking rrdtool.so.8.0.1 to rrdtool.so.4 and everything seems stable (for now). 

[lmori]

That won't work, v4 and v8 have different entrypoints. I am wondering why your install is looking for librrd.4.so that is not shipped with 18.04. Did you install using ansible? or did you have a previous install using the binary packages?

Luigi

[alchen]

Was there a fix to this issue?  I'm running into this exact issue on a fresh install of Minemeld on Ubuntu 18.04.  minemeld-web.log shows the following:

  import rrdtool

ImportError: librrd.so.4: cannot open shared object file: No such file or directory

[2019-04-11 10:32:25 +0000] [4653] [INFO] Worker exiting (pid: 1748)

[2019-04-11 10:32:25 +0000] [4650] [INFO] Shutting down: Master

[2019-04-11 10:32:25 +0000] [4650] [INFO] Reason: Worker failed to boot.

minemeld-web always shows FATAL, Exited too quickly (process log may have details).

Any thoughts?  Thanks in advance.

[Alex_Lora]

I have the same issues but on Ubuntu 14:04. Good luck finding an answer. 

[Carlos_Gomes]

i too have this issue. Installed Ubuntu 18.04 then followed the minemeld-ansible instructions. it was working. i restarted the server and it stopped working. Minemeld-web Fatal. 

[2019-10-31 09:33:18 +0000] [4698] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
worker.init_process()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
super(GeventWorker, self).init_process()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module>
app = create_app()
File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 67, in create_app
from . import metricsapi # noqa
File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 19, in <module>
import rrdtool
ImportError: librrd.so.4: cannot open shared object file: No such file or directory

what i dont get is that after installation it started and was working fine then rebooted and issue.

[Carlos_Gomes]

@bsmith2 how did you create the sym link? I cannot seem to get my head around this.

[a.jones]

I have found if you install on Ubuntu 18.04 Bionic Beaver server edition it works. it took me two installs with reboots to get it working but I get a log in. I think the first install missed some things. Second install, rerun from the beginning as if a fresh install.

regards

Adrian

[Carlos_Gomes]

Thank you @a.jones i will try as you suggested to see if that works.