Minemeld-web FATAL (Ubuntu 18.04)

[LassiK]

Hi

Ive been having a sweaty couple of days trying to get this to work and I must say ubuntu is not my forte.

Anyway this all started from Ubuntu update. We had 14.04 which had sudo vulnerability and I was tasked to update this OS.

Only thing running on it was this Minemeld and ofcourse after update it broke. 

I think Ive finally got partly working, but subject is still broken. Ive been trying this forum and internet to fix without progress so far.

If I try to login from webpage it gives me error [ERROR CHECKING CREDENTIALS: BAD GATEWAY]

I check services: 

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf statusminemeld-engine RUNNING pid 3511, uptime 0:12:51
minemeld-supervisord-listener RUNNING pid 3510, uptime 0:12:51
minemeld-traced RUNNING pid 3512, uptime 0:12:51
minemeld-web FATAL Exited too quickly (process log may have details)

So this here is the problem right? There was even a time when I was able to get this minemeld-web RUNNING after I did minemeld update, but then I lost webpage to 404 error....

1. anyway here is the minemeld-web log as I have no idea how to read this:

less /opt/minemeld/log/minemeld-web.log
[2018-11-22 11:53:07 +0000] [12237] [INFO] Starting gunicorn 19.5.0
[2018-11-22 11:53:07 +0000] [12237] [INFO] Listening at: http://127.0.0.1:5000 (12237)
[2018-11-22 11:53:07 +0000] [12237] [INFO] Using worker: gevent
[2018-11-22 11:53:07 +0000] [12271] [INFO] Booting worker with pid: 12271
[2018-11-22 11:53:07 UTC] [12271] [INFO] MMLogger started
/opt/minemeld/engine/0.9.50.post4/local/lib/python2.7/site-packages/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead.
import flask.ext.login
/opt/minemeld/engine/0.9.50.post4/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.DeprecatedIn23,
[2018-11-22 11:53:08 UTC] [12271] [INFO] Config loaded: {'MINEMELD_LOCAL_LIBRARY_PATH': '/opt/minemeld/local/library', 'USERS_DB': 'wsgi.htpasswd', 'MINEMELD_LOCAL_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes', 'SUPERVISOR_URL': 'unix:///var/run/minemeld/minemeld.sock', 'MINEMELD_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes/:/opt/minemeld/prototypes/current', 'MINEMELD_PIP_PATH': '/opt/minemeld/engine/current/bin/pip', 'MINEMELD_TRACED_PURGE_PATH': '/opt/minemeld/engine/current/bin/mm-traced-purge', 'API_AUTH_ENABLED': True, 'MINEMELD_LOG_DIRECTORY_PATH': '/opt/minemeld/log', 'MINEMELD_7Z_PATH': '/usr/bin/7z', 'MINEMELD_RESTORE_PATH': '/opt/minemeld/engine/current/bin/mm-restore', 'MINEMELD_LOCAL_PATH': '/opt/minemeld/local', 'MINEMELD_GIT_PATH': '/usr/bin/git', 'SNS_ENABLED': True}
[2018-11-22 11:53:08 UTC] [12271] [INFO] USERS_DB loaded from /opt/minemeld/local/config/api/wsgi.htpasswd
[2018-11-22 11:53:08 UTC] [12271] [INFO] FEEDS_USERS_DB loaded from /opt/minemeld/local/config/api/feeds.htpasswd
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] New uuid file created.
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] Instance uuid = 9548c961933c4e9f9298e99cd6be7a65
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] MineMeld cloud notification service is ready.
[2018-11-22 11:53:09 UTC] [12271] [INFO] Extensions frigidaire not found, paths not loaded
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/supervisor/minemeld-engine/restart' (HEAD, OPTIONS, GET) -> supervisor.restart_minemeld_engine>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/supervisor/minemeld-engine/start' (HEAD, OPTIONS, GET) -> supervisor.start_minemeld_engine>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/supervisor/minemeld-engine/stop' (HEAD, OPTIONS, GET) -> supervisor.stop_minemeld_engine>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/supervisor/minemeld-web/hup' (HEAD, OPTIONS, GET) -> supervisor.hup_minemeld_web>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/events/status' (HEAD, OPTIONS, GET) -> status.get_status_events>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/backup/import' (POST, OPTIONS) -> status.import_local_backup>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/users/current' (HEAD, OPTIONS, GET) -> aaa.get_current_user>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions/git-install' (POST, OPTIONS) -> extensions.install_from_git>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions/git-refs' (HEAD, OPTIONS, GET) -> extensions.get_git_refs>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/validate/syslogminerrule' (POST, OPTIONS) -> validate.validate_syslogminerrule>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/metrics/minemeld' (HEAD, OPTIONS, GET) -> metrics.get_global_metrics>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/committed' (HEAD, OPTIONS, GET) -> config.get_committed_config>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/traced/purge-all' (HEAD, OPTIONS, GET) -> traced.traced_purge_all>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/minemeld' (HEAD, OPTIONS, GET) -> status.get_minemeld_status>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/running' (HEAD, OPTIONS, GET) -> config.get_running_config>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/mgmtbus' (HEAD, OPTIONS, GET) -> config.get_mgmtbus>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/system' (HEAD, OPTIONS, GET) -> status.get_system_status>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/config' (HEAD, OPTIONS, GET) -> status.get_minemeld_running_config>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/backup' (POST, OPTIONS) -> status.generate_local_backup>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/mkwish' (POST, OPTIONS) -> status.sns_wish>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/reload' (HEAD, OPTIONS, GET) -> config.reload_running_config>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/commit' (POST, OPTIONS) -> config.commit>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/fabric' (HEAD, OPTIONS, GET) -> config.get_fabric>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/traced/query' (HEAD, OPTIONS, GET) -> traced.traced_query>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/info' (HEAD, OPTIONS, GET) -> status.get_system_info>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/info' (HEAD, OPTIONS, GET) -> config.get_config_info>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/full' (HEAD, OPTIONS, GET) -> config.get_config_full>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/node' (POST, OPTIONS) -> config.create_node>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/logs/minemeld-engine.log' (HEAD, OPTIONS, GET) -> logs.get_minemeld_engine_log>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/logs/minemeld-web.log' (HEAD, OPTIONS, GET) -> logs.get_minemeld_web_log>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/feeds' (HEAD, OPTIONS, GET) -> aaa.get_feeds>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/tags' (HEAD, OPTIONS, GET) -> aaa.get_tags>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/taxii-collection-management-service' (POST, OPTIONS) -> taxiicollmgmt.taxii_collection_mgmt_service>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/taxii-discovery-service' (POST, OPTIONS) -> taxiidiscovery.taxii_discovery_service>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/taxii-poll-service' (POST, OPTIONS) -> taxiipoll.taxii_poll_service>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/supervisor' (HEAD, OPTIONS, GET) -> supervisor.service_status>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions' (HEAD, OPTIONS, GET) -> extensions.list_extensions>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions' (POST, OPTIONS) -> extensions.upload_extension>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/prototype' (HEAD, OPTIONS, GET) -> prototype.list_prototypes>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/metrics/' (HEAD, OPTIONS, GET) -> metrics.get_metrics>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/logout' (HEAD, OPTIONS, GET) -> login.logout>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/login' (HEAD, POST, OPTIONS, GET) -> login.login>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/backup/import/<backup_id>/restore' (POST, OPTIONS) -> status.restore_local_backup>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/users/<subsystem>/<username>/attributes' (POST, OPTIONS) -> aaa.set_user_attributes>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/events/query/<quuid>' (HEAD, OPTIONS, GET) -> status.get_query_events>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/traced/query/<query_uuid>/kill' (HEAD, OPTIONS, GET) -> traced.traced_kill_query>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/data/<datafilename>/append' (POST, OPTIONS) -> configdata.append_config_data>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/<nodename>/signal/<signalname>' (HEAD, POST, OPTIONS, GET) -> status.signal_node>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/jobs/<job_group>/<jobid>/log' (HEAD, OPTIONS, GET) -> jobs.get_job_log>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/feeds/<feedname>/attributes' (PUT, POST, OPTIONS) -> aaa.set_feed_attributes>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/users/<subsystem>/<username>' (PUT, OPTIONS) -> aaa.set_user_password>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/aaa/users/<subsystem>/<username>' (OPTIONS, DELETE) -> aaa.delete_user>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions/<extension>/deactivate' (HEAD, POST, OPTIONS, GET) -> extensions.deactivate_extension>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions/<extension>/uninstall' (HEAD, POST, OPTIONS, GET) -> extensions.uninstall_extension>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/extensions/<extension>/activate' (POST, OPTIONS) -> extensions.activate_extension>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/metrics/minemeld/<nodetype>' (HEAD, OPTIONS, GET) -> metrics.get_node_type_metrics>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/metrics/<node>/<metric>' (HEAD, OPTIONS, GET) -> metrics.get_metric>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/status/backup/<jobid>' (HEAD, OPTIONS, GET) -> status.get_local_backup>
[2018-11-22 11:53:09 UTC] [12271] [DEBUG] app rule: <Rule '/config/node/<nodenum>' (HEAD, OPTIONS, GET) -> config.get_node>

2. Should this path have a config/ folder also?

/opt/minemeld/
engine/ local/ log/ prototypes/ supervisor/ www/

Anything else?

[LassiK]

UPDATE:

Ofcourse it starts working right after you ask help... well "working"

After following this thread and doing these python RDDtool stuff, every service is running and I can access webpage.

Although something is still broken because when I do external dynamic list -> test URL it says access error.... more work, but its getting close

[lmori · Accepted solution]

Hi @LassiK ,

MineMeld binaries are not compatible with Ubuntu 18.04.

My suggestion for running it on 18.04 is using Docker image. Moving forward that will be the preferred way of deployment.

Luigi