Reply
Highlighted
L1 Bithead

Re: Ubuntu 18.04 install errors

Seeing similar issues on fresh 18.04 Ubuntu Server LTS install here too ( using Ansible method which usually works great! thanks!)

I have another 18.04 Ubuntu Server LTS image for home (18.04.2) that seems to be working ok but not updated recently.
But doing a brand new Ubuntu Server LTS build at work does not fire up minemeld.

I am guessing some package update has broken something or something is out of date with newer package dependancy as i have updated the Ubuntu base OS before doing any minemeld Ansible work.

Current version on Non-Working version is 18.04.3.

There is concern with the build also around use of Python 2.7 due to depreciation just now in Jan2020.

"DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support"

Will there be a python 3 build of Minemeld in the future?

 

Status shows a buffer overflow with this minemeld status command.

 

user@server:~/minemeld-ansible$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf maintail ent buffer overflowed, discarding event 18 2020-01-12 23:09:58,557 INFO exited: minemeld-web (exit status 3; not expected) 2020-01-12 23:09:58,557 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 19 2020-01-12 23:09:58,557 INFO gave up: minemeld-web entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,580 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 20 2020-01-12 23:09:58,580 INFO exited: minemeld-supervisord-listener (exit status 1; not expected) 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 21 2020-01-12 23:09:58,581 INFO gave up: minemeld-supervisord-listener entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,581 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 22 2020-01-12 23:09:58,581 INFO exited: minemeld-traced (exit status 1; not expected) 2020-01-12 23:09:58,600 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 23 2020-01-12 23:09:58,601 INFO gave up: minemeld-traced entered FATAL state, too many start retries too quickly 2020-01-12 23:09:58,617 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 24 2020-01-12 23:09:58,617 INFO exited: minemeld-engine (exit status 1; not expected) 2020-01-12 23:09:59,618 ERRO pool minemeld-supervisord-listener event buffer overflowed, discarding event 25 2020-01-12 23:09:59,618 INFO gave up: minemeld-engine entered FATAL state, too many start retries too quickly

 

Checking supervisor Log shows more errors in packaging.

 

user@server :/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources$ tail -200 /opt/minemeld/log/minemeld-supervisord-listener.log module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked Traceback (most recent call last): File "/opt/minemeld/engine/current/bin/mm-supervisord-listener", line 11, in load_entry_point('minemeld-core', 'console_scripts', 'mm-supervisord-listener')() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point return ep.load() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load return self.resolve() File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/opt/minemeld/engine/core/minemeld/supervisord/listener.py", line 7, in import ujson ImportError: /opt/minemeld/engine/current/local/lib/python2.7/site-packages/ujson.so: undefined symbol: Buffer_AppendShortHexUnchecked

 

The two packages i think might be at fault are (IMO I'm no python guru).

ujson

load_entry_point

 

Any clues on getting Minemeld to work on latest 18.04.3+ using manual work around? And to maintainers of minemeld what are the plans for migration to Python 3?

 

Thanks for providing a great product that helps us to automate black and whitelists with ease.

L0 Member

Re: Ubuntu 18.04 install errors

It's seems that I'm having the same problem as you... I upgrade my Ubuntu and updated Minemeld and it's giving the error with the ujson.so file..

 

Wouldn't want to switch distro to use Minemeld...

Highlighted
L0 Member

Re: Ubuntu 18.04 install errors

Hi,

 

we found a work-around to get it to work on Ubuntu 18.04.4 LTS.

Install the package python-ujson with apt:

sudo apt install python-ujson

 Then move the ujson.so file to usjon.old in /opt/minemeld/engine/current/local/lib/python2.7/site-packages

ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ mv ujson.so ujson.so.old

 Link the ujson.so from the apt package

ubuntu@lxminemeld:/opt/minemeld/engine/current/local/lib/python2.7/site-packages$ ln -s /usr/lib/python2.7/dist-packages/ujson.x86_64-linux-gnu.so ujson.so

 

Restart everything

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/sup
ervisor/config/supervisord.conf restart all

 

Good luck!

 

Kind Regards,

Edd

Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

Yes yes yes i got it working too thank you @EdwardMarshall for your input.

Now i was also having issues with the rrdtool so i followed the same method as you sent but for the rrdtool.

sudo apt install python-rrdtool

Followed by

cd /opt/minemeld/engine/current/local/lib/python2.7/site-packages

Then

mv rrdtool.so rrdtool.so.old

Finally

ln -s /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu.so rrdtool.so

 

Restart Minemeld

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf start

 

If you have issues with the minemeld web or minemeld engine starting and you check the logs and see the errors on flask.ext.login being depreciated use flask_login. find and replace flask.ext.login with flask_login on the aaa.py file under /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/flask

 

good luck

Carlos

Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

Just to add one more thing, Pulling data from taxiclient with AlienVault i get an sslv3 handshake failure. I did try to update gevent and greenlet same ways we did the rrdtool and ujson to no avail. it seems that MineMeld creators need to update this to a later version of python also since 2.7 is old and deprecated i believe.

Carlos_Gomes_0-1581007559087.png

 

Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

and finally got this working.... used pycharm to figure out where the errors were.

Install newer version of libtaxii by doing:

pip install libtaxii --force

here is the log of that install:

root@:/opt/minemeld/engine/current/local# pip install libtaxii --force
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting libtaxii
Downloading libtaxii-1.1.115-py2.py3-none-any.whl (130 kB)
|████████████████████████████████| 130 kB 4.6 MB/s
Collecting python-dateutil>=1.4.1
Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
|████████████████████████████████| 227 kB 8.2 MB/s
Collecting six>=1.9.0
Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting lxml>=2.2.3
Downloading lxml-4.5.0-cp27-cp27mu-manylinux1_x86_64.whl (5.7 MB)
|████████████████████████████████| 5.7 MB 9.0 MB/s
ERROR: minemeld-core 0.9.66 has requirement libtaxii==1.1.107, but you'll have libtaxii 1.1.115 which is incompatible.
ERROR: minemeld-core 0.9.66 has requirement lxml==4.1.0, but you'll have lxml 4.5.0 which is incompatible.
ERROR: minemeld-core 0.9.66 has requirement six==1.11.0, but you'll have six 1.14.0 which is incompatible.

 

you need to update the following file

/opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld_core-0.9.66.dist-info/metadata

 

change libtaxii version to whatever the newest is, in my case it was 1.1.115, then lxml to 4.5.0 and six to 1.14.0 so it looks like:

Requires-Dist: six (==1.14.0)
Requires-Dist: lxml (==4.5.0)

Requires-Dist: libtaxii (==1.1.115)

 

then restart minemeld.

 

 

 

 

Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

**bleep** just as i thought i had this fixed no classes now load... COMMIT FAILED: Class minemeld.ft.ipop.AggregateIPv4FT in IP_Aggregator not safe to load

 

any class is failing.

Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

and a simple full reboot fixed this issue. Finally a working product. Too many changes needed to get minemeld-core working in a stable way. Would be great to have the guys who developed this to move to a supported version of python. Either way being that its an open source product and everyone else has their jobs one cannot ask for much more. hopefully i was able to contribute to the product with my findings.

Highlighted
L1 Bithead

Re: Ubuntu 18.04 install errors

Thanks @EdwardMarshall @Carlos_Gomes for your work on getting Ubuntu 18.04.3 working somewhat..... As i was setting this up to show work i didn't have the time to investigate further so went the 'Docker' route which just worked fine! It would be nice to have some comment from Palo Minemeld Devs on what future plans are for the product especially a migration to a supported python version? I'm not a developer and can't work on contributions myself so thanks so far for all the great work done on providing a nice product for extending security on our firewalls and other security devices to ingest, de-dupe/aggregate and output data for those products.....looking forward to what plans are happening to extend this great product.
Highlighted
L3 Networker

Re: Ubuntu 18.04 install errors

Thank you @Paul_Stinson after doing all of this work now the engine just stops I'm still trying to see what is causing it. If it worked for you on Docker which I never checked if there was a guide for this, is there one that you followed? My installation after all the small fixes works but once I added some alien vault 3red party vendor threat feeds it broke. I'm certain removing the taxi client feeds will allow it the engine to run stable but if I cannot get the lists from those feeds what's the point right? And I agree with you it would be good to hear from the dev's on this I even posted questions on github too no answers... it's a pity there aren't so many contributing to what I think is a good product. I too am no developer or even have any python background.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!