About Carlos_Gomes

another update i was able to follow my own notes to update the libtaxii within the docker container, i would think that if the container stops running or the linux vm restarts the changes i made will need to be redone. i have yet to test this.  ... View more

These are the errors i get. I know on minemeld core i fix it by upgrading the libtaxii to latest version but this is in docker so no idea how to do it here.   2020-02-10T17:38:23 (153)basepoller._polling_loop INFO: Polling AlienVault_Any_Miner 2020-02-10T17:38:23 (153)basepoller._poll ERROR: Exception in polling loop for AlienVault_Any_Miner: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)> Traceback (most recent call last): File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 737, in _poll performed = self._polling_loop() File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 584, in _polling_loop iterator = self._build_iterator(now) File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 1131, in _build_iterator self._discover_services(tc) File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 292, in _discover_services resp = self._call_taxii_service(self.discovery_service, tc, request) File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 282, in _call_taxii_service port=port File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/libtaxii/clients.py", line 337, in call_taxii_service2 response = urllib.request.urlopen(req) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/opt/minemeld/engine/0.9.66/local/lib/python2.7/site-packages/libtaxii/clients.py", line 363, in https_open return self.do_open(self.get_connection, req) File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open raise URLError(err) URLError: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:590)> ... View more

@Paul_Stinson so i tested the docker version and the same issues out of the box apply when connecting a miner to Alienvault... so annoying this whole thing. only minemeld-ansible release works with it but it doesnt stay up for longer than 24hours. This product is so buggy for me. i wish it just worked. ... View more

Thank you @Paul_Stinson this is helpful, i will try to see if with the docker version minemeld core works and is stable. Minemeld-ansible version stops working for me after 24 hours or so and i then need to re-run the playbook so it is not sustainable but it didn't need many of those fixes we had to do to minemeld core version which is strange in itself. However i don't want to have to run the playbook every 24 hours, or have to write some sort of script that monitors when its broken to then run the playbook. And like you i agree too, python2.7 being deprecated and this being a sec tool it should be running in the latest versions for better support and security. Hopefully the minemeld developers have been quiet because they are all working away for the community in building a better version and more secure. Time will tell... thanks again. ... View more

Thank you @Paul_Stinson after doing all of this work now the engine just stops I'm still trying to see what is causing it. If it worked for you on Docker which I never checked if there was a guide for this, is there one that you followed? My installation after all the small fixes works but once I added some alien vault 3red party vendor threat feeds it broke. I'm certain removing the taxi client feeds will allow it the engine to run stable but if I cannot get the lists from those feeds what's the point right? And I agree with you it would be good to hear from the dev's on this I even posted questions on github too no answers... it's a pity there aren't so many contributing to what I think is a good product. I too am no developer or even have any python background. ... View more

and a simple full reboot fixed this issue. Finally a working product. Too many changes needed to get minemeld-core working in a stable way. Would be great to have the guys who developed this to move to a supported version of python. Either way being that its an open source product and everyone else has their jobs one cannot ask for much more. hopefully i was able to contribute to the product with my findings. ... View more

**bleep** just as i thought i had this fixed no classes now load...  COMMIT FAILED: Class minemeld.ft.ipop.AggregateIPv4FT in IP_Aggregator not safe to load   any class is failing. ... View more

and finally got this working.... used pycharm to figure out where the errors were. Install newer version of libtaxii by doing: pip install libtaxii --force here is the log of that install: root@:/opt/minemeld/engine/current/local# pip install libtaxii --force DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Collecting libtaxii Downloading libtaxii-1.1.115-py2.py3-none-any.whl (130 kB) |████████████████████████████████| 130 kB 4.6 MB/s Collecting python-dateutil>=1.4.1 Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) |████████████████████████████████| 227 kB 8.2 MB/s Collecting six>=1.9.0 Downloading six-1.14.0-py2.py3-none-any.whl (10 kB) Collecting lxml>=2.2.3 Downloading lxml-4.5.0-cp27-cp27mu-manylinux1_x86_64.whl (5.7 MB) |████████████████████████████████| 5.7 MB 9.0 MB/s ERROR: minemeld-core 0.9.66 has requirement libtaxii==1.1.107, but you'll have libtaxii 1.1.115 which is incompatible. ERROR: minemeld-core 0.9.66 has requirement lxml==4.1.0, but you'll have lxml 4.5.0 which is incompatible. ERROR: minemeld-core 0.9.66 has requirement six==1.11.0, but you'll have six 1.14.0 which is incompatible.   you need to update the following file /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld_core-0.9.66.dist-info/metadata   change libtaxii version to whatever the newest is, in my case it was 1.1.115, then lxml to 4.5.0 and six to 1.14.0 so it looks like: Requires-Dist: six (==1.14.0) Requires-Dist: lxml (==4.5.0) Requires-Dist: libtaxii (==1.1.115)   then restart minemeld.         ... View more

Just to add one more thing, Pulling data from taxiclient with AlienVault i get an sslv3 handshake failure. I did try to update gevent and greenlet same ways we did the rrdtool and ujson to no avail. it seems that MineMeld creators need to update this to a later version of python also since 2.7 is old and deprecated i believe.   ... View more

Yes yes yes i got it working too thank you @EdwardMarshall for your input. Now i was also having issues with the rrdtool so i followed the same method as you sent but for the rrdtool. sudo apt install python-rrdtool Followed by cd /opt/minemeld/engine/current/local/lib/python2.7/site-packages Then mv rrdtool.so rrdtool.so.old Finally ln -s /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu.so rrdtool.so   Restart Minemeld sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf start   If you have issues with the minemeld web or minemeld engine starting and you check the logs and see the errors on flask.ext.login being depreciated use flask_login. find and replace flask.ext.login with flask_login on the aaa.py file under /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/flask   good luck Carlos ... View more

so odd @a.jones i have tried this twice, i have rebuilt the server too everytime there are issues. aaa.py reports issues with the flask being deprecated and the web service cannot start, change that to flask_login and then it does but the engine never starts as it has issues with the rrdtool. I have been able to fix that since by installing it on the virtual env but out of the box this minemeld at least for me doesn't run properly. When i do get it working i still have issues connecting to an Alienvault OTX pulse of mine. Complains with sslv3 handshake errors. other sources work fine. Support for this obviously open source so i have to rely on this community and that of github to help. ... View more

@a.jones correct i do too but when i try to login it fails. Can you login and is all working as expected? ... View more

Hi Adrian, i know you said you got a login but were you actually able to login? the minemeld-web service is fatal and doesnt start. I noticed in the logs about flask_login so i changed as recommend in the logs and still fails. /opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead. import flask.ext.login So after changing the flask_login and saving the aaa.py file it then no longer errors out but fails next step on loading the rrdtool.   file "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/minemeld/flask/metricsapi.py", line 19, in <module> import rrdtool ImportError: librrd.so.4: cannot open shared object file: No such file or directory   so still doesn't work. ... View more

Thank you @a.jones i will try as you suggested to see if that works. ... View more

@bsmith2 how did you create the sym link? I cannot seem to get my head around this. ... View more

@folmer i ended up uninstalling minemeld, upgraded to Ubuntu 18.04 and then deployed minemeld-ansible instead or minemeld-core now working a treat with cabby needed. sslv3 handshake errors gone. i cannot start minemeld-web service but that is a different issue altogether for another post. thank you for your reply and help. ... View more

i too have this issue. Installed Ubuntu 18.04 then followed the minemeld-ansible instructions. it was working. i restarted the server and it stopped working. Minemeld-web Fatal.  [2019-10-31 09:33:18 +0000] [4698] [ERROR] Exception in worker process Traceback (most recent call last): File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker worker.init_process() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process super(GeventWorker, self).init_process() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process self.load_wsgi() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi self.wsgi = self.app.wsgi() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load return self.load_wsgiapp() File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app __import__(module) File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module> app = create_app() File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 67, in create_app from . import metricsapi # noqa File "/opt/minemeld/engine/core/minemeld/flask/metricsapi.py", line 19, in <module> import rrdtool ImportError: librrd.so.4: cannot open shared object file: No such file or directory   what i dont get is that after installation it started and was working fine then rebooted and issue. ... View more

@folmer Correct. i actually didnt have the folder core under opt/minemeld/engine/ so i created to match your instructions. I created the requirements.txt as per minemeld file in their github and changed the requirements for pytz to pytz==2019.3 and libtaxii to libtaxii==1.1.114, i think below its what you meant to write. The errors in minemeld.engine log below as example i get that for every prototype. (26093)loader._initialize_entry_point_group ERROR: minemeld.ft.taxii.TaxiiClient not loadable: pytz==2019.3 not compatible with pytz==2015.4, libtaxii==1.1.114 not compatible with libtaxii==1.1.107   I can try this again but havent had much luck leveraging cabby, i was try to do that as with AlienVault OTX i am getting sslv3 handshake failures.   (26093)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Class minemeld.ft.taxii.TaxiiClient in Cyrebro10_OTX_Pulses not safe to load   ... View more

Hi @folmer i tried to get this going as according to your github instructions and doesn't work, i get a lot of errors. For example: oader._initialize_entry_point_group ERROR: minemeld.ft.local.YamlURLFT not loadable: pytz==2019.3 not compatible with pytz==2015.4, libtaxii==1.1.114 not compatible with libtaxii==1.1.107   also the engine now is FATAL and doesnt load properly and i do not see this as an available prototype. ... View more

It seems like you would be on your own as it is not supported. I find the community of minemeld is very small and therefore lack of responses.  ... View more

I do not think that you are running a supported release. I am running Ubuntu 16.04.6 LTS which i believe was the latest approved. You need to follow this article. https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336 ... View more

Use the pem and key files instead. ... View more

i do not think those are arbitrary. I am too running same version and with this config do not have a problem. just click on that prototype you have and click NEW then change the config, save, then add new node using that prototype and you know the rest. will take you 5 minutes. There arent many people it seems to me reading these. I have my question posted on 4 different links and no response on stripping http/https from the taxiclient feed.  ... View more

So if i were i would setup exactly as i have as you do not need minemeld on the source and collection names. TAGS: ConfidenceHigh ShareLevelRed   CONFIG: age_out: default: last_seen+30d sudden_death: false attributes: confidence: 30 share_level: red client_cert_required: true collection: username.FSISAC_FEED discovery_service: https://analysis.fsisac.com/taxii-discovery-service initial_interval: 90d source_name: fs-isac.username.FSISAC_FEED ... View more

Can you share what your prototype config looks like, here is mine. This is working for me without any problems. ... View more