- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
04-10-2019 09:52 AM
Hi There,
Running into issues trying to install on Ubuntu 18.04
During the install process at the end I get this message:
TASK [minemeld : requirements] ******************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": "/opt/minemeld/engine/current/bin/pip2 freeze", "msg": "\n:stderr: Traceback (most recent call last):\n File \"/opt/minemeld/engine/current/bin/pip2\", line 6, in <module>\n from pip._internal import main\n File \"/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pip/_internal/__init__.py\", line 5, in <module>\n import logging\n File \"/usr/lib/python2.7/logging/__init__.py\", line 26, in <module>\n import sys, os, time, cStringIO, traceback, warnings, weakref, collections\n File \"/usr/lib/python2.7/weakref.py\", line 14, in <module>\n from _weakref import (\nImportError: cannot import name _remove_dead_weakref\n"}
to retry, use: --limit @/home/timpo/minemeld-ansible/local.retry
And this is the output of journalctl -xe:
-- Unit minemeld.service has begun starting up.
Apr 10 12:40:43 ise-miner mkdir[16946]: /bin/mkdir: cannot create directory ‘/var/run/minemeld’: File exists
Apr 10 12:40:44 ise-miner supervisord[16948]: Traceback (most recent call last):
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/current/bin/supervisord", line 6, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.supervisord import main
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisord.py", line 41, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: from supervisor.options import ServerOptions
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: import pkg_resources
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
Apr 10 12:40:44 ise-miner supervisord[16948]: __import__('pkg_resources.extern.packaging.requirements')
Apr 10 12:40:44 ise-miner supervisord[16948]: File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
Apr 10 12:40:44 ise-miner supervisord[16948]: "distribution.".format(**locals())
Apr 10 12:40:44 ise-miner supervisord[16948]: ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the p
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Control process exited, code=exited status=1
Apr 10 12:40:44 ise-miner systemd[1]: minemeld.service: Failed with result 'exit-code'.
Apr 10 12:40:44 ise-miner sudo[16924]: pam_unix(sudo:session): session closed for user root
Apr 10 12:40:44 ise-miner systemd[1]: Failed to start Process Monitoring and Control Daemon.
And here's the output of sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status:
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/supervisorctl", line 6, in <module>
from supervisor.supervisorctl import main
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/supervisorctl.py", line 36, in <module>
from supervisor.options import ClientOptions
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/supervisor/options.py", line 15, in <module>
import pkg_resources
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 84, in <module>
__import__('pkg_resources.extern.packaging.requirements')
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/pkg_resources/extern/__init__.py", line 61, in load_module
"distribution.".format(**locals())
ImportError: The 'packaging.requirements' package is required; normally this is bundled with this package so if you get this warning, consult the packager of your distribution.
Any ideas?
12-04-2019 12:45 PM
Hi Adrian, i know you said you got a login but were you actually able to login? the minemeld-web service is fatal and doesnt start. I noticed in the logs about flask_login so i changed as recommend in the logs and still fails.
/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead.
import flask.ext.login
So after changing the flask_login and saving the aaa.py file it then no longer errors out but fails next step on loading the rrdtool.
file "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/minemeld/flask/metricsapi.py", line 19, in <module>
import rrdtool
ImportError: librrd.so.4: cannot open shared object file: No such file or directory
so still doesn't work.
12-09-2019 08:50 AM
I got a login using the default Iogin credntials.
12-09-2019 08:54 AM
@a.jones correct i do too but when i try to login it fails. Can you login and is all working as expected?
12-09-2019 08:57 AM
Yes, I log in successfully.
12-09-2019 09:04 AM
so odd @a.jones i have tried this twice, i have rebuilt the server too everytime there are issues. aaa.py reports issues with the flask being deprecated and the web service cannot start, change that to flask_login and then it does but the engine never starts as it has issues with the rrdtool. I have been able to fix that since by installing it on the virtual env but out of the box this minemeld at least for me doesn't run properly. When i do get it working i still have issues connecting to an Alienvault OTX pulse of mine. Complains with sslv3 handshake errors. other sources work fine. Support for this obviously open source so i have to rely on this community and that of github to help.
12-13-2019 12:29 AM
Hi Carlos,
I have just rebuilt this solution as a test and I can guarantee it works. Try it on a VMware workstation build first. You probably need to do a fresh install (not upgrade an existing solution). Use Ubuntu 18.04 Bionic Beaver and follow the Ansible setup. On the VMWare Workstation I had to follow the install process twice but I think some updates didn't install. On the live virtual server install, it worked first time.
Download Ubuntu Server from:
: http://releases.ubuntu.com/18.04/ - the Bionic Beaver version.
Anisible Install:
https://github.com/PaloAltoNetworks/minemeld-ansible
See how that goes. Remember do a fresh install.
01-12-2020 04:05 PM - edited 01-12-2020 04:11 PM
Seeing similar issues on fresh 18.04 Ubuntu Server LTS install here too ( using Ansible method which usually works great! thanks!)
I have another 18.04 Ubuntu Server LTS image for home (18.04.2) that seems to be working ok but not updated recently.
But doing a brand new Ubuntu Server LTS build at work does not fire up minemeld.
I am guessing some package update has broken something or something is out of date with newer package dependancy as i have updated the Ubuntu base OS before doing any minemeld Ansible work.
Current version on Non-Working version is 18.04.3.
There is concern with the build also around use of Python 2.7 due to depreciation just now in Jan2020.
"DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support"
Will there be a python 3 build of Minemeld in the future?
Status shows a buffer overflow with this minemeld status command.
user@server:~/minemeld-ansible$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf maintail ent buffer overflowed, discarding event 18 2020-01-12 23:09:58,557 INFO exited: minemeld-web (exit status 3; not expected) 2020-01-12 23:09:58,557 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 19 2020-01-12 23:09:58,557 INFO gave up: minemeld-web entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,580 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 20 2020-01-12 23:09:58,580 INFO exited: minemeld-supervisord-listener (exit status 1; not expected) 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 21 2020-01-12 23:09:58,581 INFO gave up: minemeld-supervisord-listener entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 22 2020-01-12 23:09:58,581 INFO exited: minemeld-traced (exit status 1; not expected) 2020-01-12 23:09:58,600 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 23 2020-01-12 23:09:58,601 INFO gave up: minemeld-traced entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,617 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 24 2020-01-12 23:09:58,617 INFO exited: minemeld-engine (exit status 1; not expected) 2020-01-12 23:09:59,618 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 25 2020-01-12 23:09:59,618 INFO gave up: minemeld-engine entered FATAL state, too many start retries too quickly
Checking supervisor Log shows more errors in packaging.
user@server :/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources$ tail -200 /opt/minemeld/log/minemeld-supervisord-listener.log module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked Traceback (most recent call last): File "/opt/minemeld/engine/current/bin/mm-supervisord-listener", line 11, in load_entry_point('minemeld-core', 'console_scripts', 'mm-supervisord-listener')() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point return ep.load() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load return self.resolve() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked
The two packages i think might be at fault are (IMO I'm no python guru).
ujson
load_entry_point
Any clues on getting Minemeld to work on latest 18.04.3+ using manual work around? And to maintainers of minemeld what are the plans for migration to Python 3?
Thanks for providing a great product that helps us to automate black and whitelists with ease.
01-30-2020 12:18 PM
It's seems that I'm having the same problem as you... I upgrade my Ubuntu and updated Minemeld and it's giving the error with the ujson.so file..
Wouldn't want to switch distro to use Minemeld...
02-05-2020 02:37 AM
Hi,
we found a work-around to get it to work on Ubuntu 18.04.4 LTS.
Install the package python-ujson with apt:
sudo apt install python-ujson
Then move the ujson.so file to usjon.old in /opt/minemeld/engine/current/local/lib/python2.7/site-packages
ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ mv ujson.so ujson.so.old
Link the ujson.so from the apt package
ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/ujson.x86_64-linux-gnu.so ujson.so
Restart everything
sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/sup
ervisor/config/supervisord.conf restart all
Good luck!
Kind Regards,
Edd
02-06-2020 06:56 AM
Yes yes yes i got it working too thank you @EdwardMarshall for your input.
Now i was also having issues with the rrdtool so i followed the same method as you sent but for the rrdtool.
sudo apt install python-rrdtool
Followed by
cd /opt/minemeld/engine/current/local/lib/python2.7/site-packages
Then
mv rrdtool.so rrdtool.so.old
Finally
ln -s /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu.so rrdtool.so
Restart Minemeld
sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf start
If you have issues with the minemeld web or minemeld engine starting and you check the logs and see the errors on flask.ext.login being depreciated use flask_login. find and replace flask.ext.login with flask_login on the aaa.py file under /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/flask
good luck
Carlos
02-06-2020 08:47 AM
Just to add one more thing, Pulling data from taxiclient with AlienVault i get an sslv3 handshake failure. I did try to update gevent and greenlet same ways we did the rrdtool and ujson to no avail. it seems that MineMeld creators need to update this to a later version of python also since 2.7 is old and deprecated i believe.
02-07-2020 09:04 AM
and finally got this working.... used pycharm to figure out where the errors were.
Install newer version of libtaxii by doing:
pip install libtaxii --force
here is the log of that install:
root@:/opt/minemeld/engine/current/local# pip install libtaxii --force
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting libtaxii
Downloading libtaxii-1.1.115-py2.py3-none-any.whl (130 kB)
|████████████████████████████████| 130 kB 4.6 MB/s
Collecting python-dateutil>=1.4.1
Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
|████████████████████████████████| 227 kB 8.2 MB/s
Collecting six>=1.9.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting lxml>=2.2.3
Downloading lxml-4.5.0-cp27-cp27mu-manylinux1_x86_64.whl (5.7 MB)
|████████████████████████████████| 5.7 MB 9.0 MB/s
ERROR: minemeld-core 0.9.66 has requirement libtaxii==1.1.107, but you'll have libtaxii 1.1.115 which is incompatible.
ERROR: minemeld-core 0.9.66 has requirement lxml==4.1.0, but you'll have lxml 4.5.0 which is incompatible.
ERROR: minemeld-core 0.9.66 has requirement six==1.11.0, but you'll have six 1.14.0 which is incompatible.
you need to update the following file
/opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld_core-0.9.66.dist-info/metadata
change libtaxii version to whatever the newest is, in my case it was 1.1.115, then lxml to 4.5.0 and six to 1.14.0 so it looks like:
Requires-Dist: six (==1.14.0)
Requires-Dist: lxml (==4.5.0)
Requires-Dist: libtaxii (==1.1.115)
then restart minemeld.
02-07-2020 09:25 AM
**bleep** just as i thought i had this fixed no classes now load... COMMIT FAILED: Class minemeld.ft.ipop.AggregateIPv4FT in IP_Aggregator not safe to load
any class is failing.
02-07-2020 11:12 AM
and a simple full reboot fixed this issue. Finally a working product. Too many changes needed to get minemeld-core working in a stable way. Would be great to have the guys who developed this to move to a supported version of python. Either way being that its an open source product and everyone else has their jobs one cannot ask for much more. hopefully i was able to contribute to the product with my findings.
02-09-2020 03:30 PM
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!