Ubuntu 18.04 install errors

[bsmith2]

Hi There, 

Running into issues trying to install on Ubuntu 18.04

During the install process at the end I get this message:

TASK [minemeld : requirements] ******************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": "/opt/minemeld/engine/current/bin/pip2 freeze", "msg": "\n:stderr: Traceback (most recent call last):\n File \"/opt/minemeld/engine/current/bin/pip2\", line 6, in <module>\n from pip._internal import main\n File \"/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pip/_internal/__init__.py\", line 5, in <module>\n import logging\n File \"/usr/lib/python2.7/logging/__init__.py\", line 26, in <module>\n import sys, os, time, cStringIO, traceback, warnings, weakref, collections\n File \"/usr/lib/python2.7/weakref.py\", line 14, in <module>\n from _weakref import (\nImportError: cannot import name _remove_dead_weakref\n"}
to retry, use: --limit @/home/timpo/minemeld-ansible/local.retry

And this is the output of journalctl -xe:

-- Unit minemeld.service has begun starting up.
Apr 10 12:40:43 ise-miner mkdir[16946]: /bin/mkdir: cannot create directory ‘/var/run/minemeld’: File exists
Apr 10 12:40:44 ise-miner supervisord[16948]: Traceback (most recent call last):
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/current/bin/supervisord", line 6, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.supervisord import main
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisord.py", line 41, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.options import ServerOptions
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: import pkg_resources
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: __import__('pkg_resources.extern.packaging.requirements')
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
Apr 10 12:40:44 ise-miner supervisord[16948]: "distribution.".format(**locals())
Apr 10 12:40:44 ise-miner supervisord[16948]: ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the p
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Control process exited, code=exited status=1
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Failed with result 'exit-code'.
Apr 10 12:40:44 ise-miner sudo[16924]: pam_unix(sudo:session): session closed for user root
Apr 10 12:40:44 ise-miner systemd[1]: Failed to start Process Monitoring and Control Daemon.

And here's the output of sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status:

Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/supervisorctl", line 6, in <module>
from supervisor.supervisorctl import main
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisorctl.py", line 36, in <module>
from supervisor.options import ClientOptions
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
import pkg_resources
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
__import__('pkg_resources.extern.packaging.requirements')
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
"distribution.".format(**locals())
ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the packager of your distribution.

Any ideas?

[a.jones]

Hi Carlos,

I have just rebuilt this solution as a test and I can guarantee it works. Try it on a VMware workstation build first. You probably need to do a fresh install (not upgrade an existing solution). Use Ubuntu 18.04 Bionic Beaver and follow the Ansible setup. On the VMWare Workstation I had to follow the install process twice but I think some updates didn't install. On the live virtual server install, it worked first time.

Download Ubuntu Server from:

: http://releases.ubuntu.com/18.04/ - the Bionic Beaver version.

Anisible Install:

https://github.com/PaloAltoNetworks/minemeld-ansible

See how that goes. Remember do a fresh install.

[Paul_Stinson]

Seeing similar issues on fresh 18.04 Ubuntu Server LTS install here too ( using Ansible method which usually works great! thanks!)

I have another 18.04 Ubuntu Server LTS image for home (18.04.2) that seems to be working ok but not updated recently.
But doing a brand new Ubuntu Server LTS build at work does not fire up minemeld.

I am guessing some package update has broken something or something is out of date with newer package dependancy as i have updated the Ubuntu base OS before doing any minemeld Ansible work.

Current version on Non-Working version is 18.04.3.

There is concern with the build also around use of Python 2.7 due to depreciation just now in Jan2020.

"DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support"

Will there be a python 3 build of Minemeld in the future?

Status shows a buffer overflow with this minemeld status command.

user@server:~/minemeld-ansible$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf maintail ent buffer overflowed, discarding event 18 2020-01-12 23:09:58,557 INFO exited: minemeld-web (exit status 3; not expected) 2020-01-12 23:09:58,557 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 19 2020-01-12 23:09:58,557 INFO gave up: minemeld-web entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,580 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 20 2020-01-12 23:09:58,580 INFO exited: minemeld-supervisord-listener (exit status 1; not expected) 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 21 2020-01-12 23:09:58,581 INFO gave up: minemeld-supervisord-listener entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 22 2020-01-12 23:09:58,581 INFO exited: minemeld-traced (exit status 1; not expected) 2020-01-12 23:09:58,600 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 23 2020-01-12 23:09:58,601 INFO gave up: minemeld-traced entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,617 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 24 2020-01-12 23:09:58,617 INFO exited: minemeld-engine (exit status 1; not expected) 2020-01-12 23:09:59,618 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 25 2020-01-12 23:09:59,618 INFO gave up: minemeld-engine entered FATAL state, too many start retries too quickly

Checking supervisor Log shows more errors in packaging.

user@server :/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources$ tail -200 /opt/minemeld/log/minemeld-supervisord-listener.log module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked Traceback (most recent call last): File "/opt/minemeld/engine/current/bin/mm-supervisord-listener", line 11, in load_entry_point('minemeld-core', 'console_scripts', 'mm-supervisord-listener')() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point return ep.load() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load return self.resolve() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked

The two packages i think might be at fault are (IMO I'm no python guru).

ujson

load_entry_point

Any clues on getting Minemeld to work on latest 18.04.3+ using manual work around? And to maintainers of minemeld what are the plans for migration to Python 3?

Thanks for providing a great product that helps us to automate black and whitelists with ease.

[philqmercier]

It's seems that I'm having the same problem as you... I upgrade my Ubuntu and updated Minemeld and it's giving the error with the ujson.so file..

Wouldn't want to switch distro to use Minemeld...

[EdwardMarshall]

Hi,

we found a work-around to get it to work on Ubuntu 18.04.4 LTS.

Install the package python-ujson with apt:

sudo apt install python-ujson

 Then move the ujson.so file to usjon.old in /opt/minemeld/engine/current/local/lib/python2.7/site-packages

ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ mv ujson.so ujson.so.old

 Link the ujson.so from the apt package

ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/ujson.x86_64-linux-gnu.so ujson.so

Restart everything

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/sup
ervisor/config/supervisord.conf restart all

Good luck!

Kind Regards,

Edd

[Carlos_Gomes]

Yes yes yes i got it working too thank you @EdwardMarshall for your input.

Now i was also having issues with the rrdtool so i followed the same method as you sent but for the rrdtool.

sudo apt install python-rrdtool

Followed by

cd /opt/minemeld/engine/current/local/lib/python2.7/site-packages

Then

mv rrdtool.so rrdtool.so.old

Finally

ln -s /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu.so rrdtool.so

Restart Minemeld

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf start

If you have issues with the minemeld web or minemeld engine starting and you check the logs and see the errors on flask.ext.login being depreciated use flask_login. find and replace flask.ext.login with flask_login on the aaa.py file under /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/flask

good luck

Carlos