New stix/taxii miner using cabby

[folmer]

I created a new stix/taxii miner for MineMeld, it can be found on github: https://github.com/mr-torgue/mmcabby.

It was created because I encountered severel problems with the default taxii miner and the ng miner. In general mmcabby is more stable because it uses cabby (from eclecticIQ developers of stix/taxii). It also contains support for certificate based authentication.

Improvements/remarks/bug notifications are appreciated.

[lmori]

Awesome! Are you planning to add UI to it?

[folmer]

Yes, that is one of the things I want to add in a future version. For now everything seems to be working well, so I don't know when I will work on it.

[Carlos_Gomes]

Hi @folmer i tried to get this going as according to your github instructions and doesn't work, i get a lot of errors. For example:

oader._initialize_entry_point_group ERROR: minemeld.ft.local.YamlURLFT not loadable: pytz==2019.3 not compatible with pytz==2015.4, libtaxii==1.1.114 not compatible with libtaxii==1.1.107

also the engine now is FATAL and doesnt load properly and i do not see this as an available prototype.

[folmer]

Hello Carlos, did you change the entries for pytz and libtaxii in requirements.txt? Usually requirements.txt contains "pytz==2015.4" and libtaxii"=="1.1.107". However for cabby to work these need to be newer versions. So the requirements have to be changed to "pytz>=2015.4" and libtaxii>="1.1.107". You can probably just restart the service and it should work. 

[Carlos_Gomes]

@folmer Correct. i actually didnt have the folder core under opt/minemeld/engine/ so i created to match your instructions. I created the requirements.txt as per minemeld file in their github and changed the requirements for pytz to pytz==2019.3 and libtaxii to libtaxii==1.1.114, i think below its what you meant to write.

The errors in minemeld.engine log below as example i get that for every prototype.

(26093)loader._initialize_entry_point_group ERROR: minemeld.ft.taxii.TaxiiClient not loadable: pytz==2019.3 not compatible with pytz==2015.4, libtaxii==1.1.114 not compatible with libtaxii==1.1.107

I can try this again but havent had much luck leveraging cabby, i was try to do that as with AlienVault OTX i am getting sslv3 handshake failures.

(26093)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Class minemeld.ft.taxii.TaxiiClient in Cyrebro10_OTX_Pulses not safe to load

[Carlos_Gomes]

@folmer i ended up uninstalling minemeld, upgraded to Ubuntu 18.04 and then deployed minemeld-ansible instead or minemeld-core

now working a treat with cabby needed. sslv3 handshake errors gone.

i cannot start minemeld-web service but that is a different issue altogether for another post.

thank you for your reply and help.

[Dave_W]

Hi @folmer. Have you had any success getting this extension to work with docker MineMeld distribution?

The engine/core directory didn't exist here.

I tried downloading the core git repo to engine/core, modifying the requirements.txt as recommended, but get errors when running "/opt/minemeld/engine/current/bin/python setup.py install".

creating build/temp.linux-x86_64-2.7/minemeld/packages/gdns
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DHAVE_NETDB_H= -I/usr/include/python2.7 -c minemeld/packages/gdns/_ares.c -o build/temp.linux-x86_64-2.7/minemeld/packages/gdns/_ares.o
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Any advice would be greatly appreciated. Thanks.

[folmer]

I solved the dependency issues with an ugly fix. It will work however.

I dont know if the developer of minemeld is reading this, but why are minemeld python requirements version specific? For example pytz needs to be version 2015.4, which is pretty old. Also requests needs to be version 2.20 and libtaxii needs to be 1.107. If == could be replaced by >= there would be less dependency issues.