Experiencing issue with MineMeld fresh installs

[vedd3r]

Hi,

Have anyone tried a fresh install of MineMeld recently? I'm getting issues with minemeld engine on Ubuntu 14 (using apt repo) and 16 (ansible). What I did was doing an apt-get update && apt-get upgrade on both version:

From the console:

minemeld-engine                  FATAL     Exited too quickly (process log may have details)
minemeld-supervisord-listener    RUNNING   pid 2129, uptime 0:32:42
minemeld-traced                  RUNNING   pid 2131, uptime 0:32:42
minemeld-web                     STARTING

This prevented me from logging into the web console.

From /opt/minemeld/log/minemeld-engine.log

/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
  utils.DeprecatedIn23,
2018-10-08T13:45:05 (1379)launcher.main INFO: Starting mm-run.py version 0.9.50.post1
2018-10-08T13:45:05 (1379)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 218, in main
    config = minemeld.run.config.load_config(args.config)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 567, in load_config
    return _load_config_from_dir(config_path)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 382, in _load_config_from_dir
    ccvalid, cconfig = _load_and_validate_config_from_file(ccpath)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 281, in _load_and_validate_config_from_file
    valid = resolve_prototypes(config)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 477, in resolve_prototypes
    prototypes_entrypoints = minemeld.loader.map(minemeld.loader.MM_PROTOTYPES_ENTRYPOINT)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/loader.py", line 116, in map
    _initialize_entry_point_group(entrypoint_group)
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/loader.py", line 66, in _initialize_entry_point_group
    installed = _installed_versions()
  File "/opt/minemeld/engine/0.9.50.post1/local/lib/python2.7/site-packages/minemeld/loader.py", line 37, in _installed_versions
    installed_dists = get_installed_distributions(local_only=False)
NameError: global name 'get_installed_distributions' is not defined

The above error message repeated a number of time.

Ansible issues:

Installing from ansible also has stopped with error on frigidaire stage - I've created an issue in github : https://github.com/PaloAltoNetworks/minemeld-ansible/issues/46

[vedd3r]

So I have a clone image of a working MineMeld on 16.04 and decided to do an apt-get update & upgrade on it too, to test whether the same problem occurred or not. Luckily the all the components were running. But when creating miners (ITCERTPA and phishtank) I got an error "__str__ returned non-string (type SysCallError).

Here's the log snippet from minemeld-engine log:

2018-10-09T15:52:09 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071524592 sudden_death
2018-10-09T15:52:09 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071524592 age_out
2018-10-09T15:52:09 (2548)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-10-09T15:52:09 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071524592 gc
2018-10-09T15:52:09 (2548)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-10-09T15:52:25 (2548)basepoller.hup INFO: Miner-ITCERTPA-URL - hup received, force polling
2018-10-09T15:52:25 (2548)basepoller._huppable_wait INFO: hup is clear: False
2018-10-09T15:52:25 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071545911 poll
2018-10-09T15:52:25 (2548)basepoller._polling_loop INFO: Polling Miner-ITCERTPA-URL
2018-10-09T15:52:25 (2548)connectionpool._new_conn INFO: Starting new HTTPS connection (1): infosec.cert-pa.it
2018-10-09T15:52:25 (2548)basepoller._poll ERROR: Exception in polling loop for Miner-ITCERTPA-URL: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 201, in _build_iterator
    **rkwargs
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)
2018-10-09T15:52:29 (2548)basepoller._polling_loop INFO: Polling Miner-ITCERTPA-URL
2018-10-09T15:52:29 (2548)connectionpool._new_conn INFO: Starting new HTTPS connection (1): infosec.cert-pa.it
2018-10-09T15:52:30 (2548)basepoller._poll ERROR: Exception in polling loop for Miner-ITCERTPA-URL: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 721, in _poll
    performed = self._polling_loop()
  File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 571, in _polling_loop
    iterator = self._build_iterator(now)
  File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 201, in _build_iterator
    **rkwargs
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)
2018-10-09T15:52:34 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071545911 sudden_death
2018-10-09T15:52:34 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071545911 age_out
2018-10-09T15:52:34 (2548)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-10-09T15:52:34 (2548)basepoller._actor_loop INFO: Miner-ITCERTPA-URL - command: 1539071545911 gc
2018-10-09T15:52:34 (2548)table._query_by_index INFO: Deleted in scan of _withdrawn: 0