topic Re: Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16 in General Topics

Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

hshawn — Thu, 21 Mar 2019 16:59:20 GMT

First post was marked as spam for some reason so here goes try #2 with a more specific title

Never been able to get a solid ansible deploy to CentOS so stuck with Ubuntu 14 manual install but....after staying up way too late last night trying to figure out why I suddenly could not log into Minemeld, discovering the engine would not start, doing a fresh install on Ubuntu 16.04LTS, confirming all services are started, and then STILL not being able to log in I am now officially asking for help becuase I must be missing something.

I have been running for several months without issues until I tried to log in yesterday. My theory is my Ubuntu 14 instance of Minemeld auto updated and broke something here is the message I get when trying to start it up:

In the UI

ERROR CHECKING CREDENTIALS: INTERNAL SERVER ERROR

In the CLI

# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status

minemeld-engine                  FATAL     Exited too quickly (process log may have details)
minemeld-supervisord-listener    RUNNING   pid 1470, uptime 12:02:55
minemeld-traced                  RUNNING   pid 9459, uptime 1:35:50
minemeld-web                     RUNNING   pid 1473, uptime 12:02:55

# service minemeld start

 * Starting: minemeld                                                                                               
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
  return pkg_resources.EntryPoint.parse("x="+spec).load(False)

# tail -n 50 /opt/minemeld/log/minemeld-engine.log

 raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
    config=config
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 35, in __init__
    super(RedisSet, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/actorbase.py", line 17, in __init__
    super(ActorBaseFT, self).__init__(*args, **kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 206, in __init__
    self.read_checkpoint()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 57, in read_checkpoint
    self.last_checkpoint = self.SR.get(self.redis_skey_chkp)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 880, in get
    return self.execute_command('GET', name)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
2019-03-21T08:26:53 (10943)launcher.main ERROR: Exception initializing graph
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
    mbusmaster.wait_for_chassis(timeout=10)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
    raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: checkpoint_graph called, checking current state
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: graph status None, checkpoint_graph ignored
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
    mbusmaster.wait_for_chassis(timeout=10)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
    raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis

will just dump the Ubuntu 14 install forever. I have no idea what the deal is here since all services are running, the install went perfectly fine without any issues, I even changed the admin password from the CLI to make sure I had a password set that I knew. Here is what I am seeing on the Ubuntu 16 install (I used the manual installation method):

In the UI

ERROR CHECKING CREDENTIALS: TIME OUT

In the CLI

# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status

minemeld-engine                  RUNNING   pid 1001, uptime 0:01:40
minemeld-supervisord-listener    RUNNING   pid 1000, uptime 0:01:40
minemeld-traced                  RUNNING   pid 1002, uptime 0:01:40
minemeld-web                     RUNNING   pid 1005, uptime 0:01:40

# tail -n 50 /opt/minemeld/log/minemeld-supervisor.log

minemeld-supervisord-listener-stdout---supervisor-2zuEqD.log
minemeld-traced.log
minemeld-traced-stderr---supervisor-dHNP8N.log
minemeld-web.log
minemeld-web.log.1
minemeld-web-stderr---supervisor-xbdkBx.log
supervisord.log
root@minemeld:/home/orthonovum# tail -n 50 /opt/minemeld/log/supervisord.log
2019-03-20 19:47:14,019 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:14,019 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:14,019 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:43,833 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2019-03-20 19:51:12,958 WARN received SIGTERM indicating exit request
2019-03-20 19:51:12,958 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die
2019-03-20 19:51:13,443 INFO stopped: minemeld-web (exit status 0)
2019-03-20 19:51:13,467 INFO stopped: minemeld-traced (exit status 0)
2019-03-20 19:51:16,485 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die
2019-03-20 19:51:18,573 INFO stopped: minemeld-engine (exit status 0)
2019-03-20 19:51:18,575 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM)
2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing
2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing
2019-03-20 19:51:18,951 INFO RPC interface 'supervisor' initialized
2019-03-20 19:51:18,951 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-03-20 19:51:18,952 INFO daemonizing the supervisord process
2019-03-20 19:51:18,953 INFO supervisord started with pid 1653
2019-03-20 19:51:19,958 INFO spawned: 'minemeld-supervisord-listener' with pid 1654
2019-03-20 19:51:19,960 INFO spawned: 'minemeld-engine' with pid 1655
2019-03-20 19:51:19,963 INFO spawned: 'minemeld-traced' with pid 1656
2019-03-20 19:51:19,971 INFO spawned: 'minemeld-web' with pid 1657
2019-03-20 19:51:20,990 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:20,990 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:20,990 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:50,263 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2019-03-20 20:03:54,120 WARN received SIGTERM indicating exit request
2019-03-20 20:03:54,121 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die
2019-03-20 20:03:54,271 INFO stopped: minemeld-web (exit status 0)
2019-03-20 20:03:54,297 INFO stopped: minemeld-traced (exit status 0)
2019-03-20 20:03:57,315 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die
2019-03-20 20:03:59,400 INFO stopped: minemeld-engine (exit status 0)
2019-03-20 20:03:59,402 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM)
2019-03-21 08:46:35,884 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing
2019-03-21 08:46:35,909 INFO RPC interface 'supervisor' initialized
2019-03-21 08:46:35,909 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-03-21 08:46:35,912 INFO daemonizing the supervisord process
2019-03-21 08:46:35,913 INFO supervisord started with pid 948
2019-03-21 08:46:36,917 INFO spawned: 'minemeld-supervisord-listener' with pid 1000
2019-03-21 08:46:36,920 INFO spawned: 'minemeld-engine' with pid 1001
2019-03-21 08:46:36,924 INFO spawned: 'minemeld-traced' with pid 1002
2019-03-21 08:46:36,936 INFO spawned: 'minemeld-web' with pid 1005
2019-03-21 08:46:38,202 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:46:38,202 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:46:38,202 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:47:07,420 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)

Hoping this is something that is easily spotted and fixed (for either Ubuntu 14 or 16) that I am just not knowing what to look for/at. Any ideas here? My EDLs are down atm

TIA

Re: Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

hshawn — Fri, 22 Mar 2019 18:03:26 GMT

Update to this. I tried the Ubuntu 16.04LTS Ansible install too. The install goes off without a hitch but I get the same issue/error/unable to log in after installing and rebooting. Just because I like to torture myself I am going to also try the CentOS install (Which I have working in a production environment but it is an older version of Minemeld) and see if I get any luck there.

I just want a working Minemeld. But I am really having to work at it 😄

UPDATE: Same thing with fresh CentOS7 ansible install. Beginning to think there is a bug somewhere... (>.>)

UPDATE#2: Looks like python2.7 is using udp/5000 and the minemeld web log is constantly complaining about the inability to connect to 127.0.0.1:5000. If I kill the python process using 5000 then I can log in, but of course its all messed up and says "ERROR RETRIEVING OUTPUT METRIX: Timeout"

So something python is running on udp port 5000 is interferring with the ability to log into the web UI but whatever this is must be a critical process so killing it is not the answer. I am past my expertise with minemeld so hopefully someone intimately familiar with the way it is designed can help out.

web log spams the following

[ERROR] Can't connect to ('127.0.0.1', 5000)

UPDATE #3: Fresh install of Ubuntu 14LTS then fresh manual install of Minemeld = same issue unable to log in. 99.99% sure at this point that it is nothing I am doing or my VMs but something in the minemeld code.

Re: Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

l0pht1337 — Tue, 23 Apr 2019 20:25:46 GMT

Hi mate,

I was experiencing the same problem. It appears that I have got arround this by killing the processes on 5000. I haven't come across any other problems after doing this.

I ran the following command:

sudo fuser -k 5000/tcp

Hope this helps, let me know how you get on...

Ryan

Re: Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

Alex_Lora — Wed, 31 Jul 2019 15:33:14 GMT

Im having the same issues as @hshawn. This was working for several weeks and now I cant log in. I get the: ERROR CHECKING CREDENTIALS: Bad Gateway error message. Here is my output.

lsb_release -r

Release:        14.04

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

minemeld-engine                  RUNNING   pid 1943, uptime 0:17:12
minemeld-supervisord-listener    RUNNING   pid 1942, uptime 0:17:12
minemeld-traced                  RUNNING   pid 1944, uptime 0:17:12
minemeld-web                     FATAL     Exited too quickly (process log may have details)

tail -n 100 /opt/minemeld/log/minemeld-web.log

[2019-07-31 15:00:54 +0000] [2252] [INFO] Shutting down: Master
[2019-07-31 15:00:54 +0000] [2252] [INFO] Reason: Worker failed to boot.
[2019-07-31 15:00:57 +0000] [2262] [INFO] Starting gunicorn 19.5.0
[2019-07-31 15:00:57 +0000] [2262] [INFO] Listening at: http://127.0.0.1:5000 (2262)
[2019-07-31 15:00:57 +0000] [2262] [INFO] Using worker: gevent
[2019-07-31 15:00:57 +0000] [2266] [INFO] Booting worker with pid: 2266
[2019-07-31 15:00:57 UTC] [2266] [INFO] MMLogger started
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead.
  import flask.ext.login
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
  utils.DeprecatedIn23,
[2019-07-31 15:00:57 UTC] [2266] [INFO] Config loaded: {'MINEMELD_LOCAL_LIBRARY_PATH': '/opt/minemeld/local/library', 'USERS_DB': 'wsgi.htpasswd', 'MINEMELD_LOCAL_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes', 'SUPERVISOR_URL': 'unix:///var/run/minemeld/minemeld.sock', 'MINEMELD_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes/:/opt/minemeld/prototypes/current', 'MINEMELD_PIP_PATH': '/opt/minemeld/engine/current/bin/pip', 'MINEMELD_TRACED_PURGE_PATH': '/opt/minemeld/engine/current/bin/mm-traced-purge', 'API_AUTH_ENABLED': True, 'MINEMELD_LOG_DIRECTORY_PATH': '/opt/minemeld/log', 'MINEMELD_7Z_PATH': '/usr/bin/7z', 'MINEMELD_RESTORE_PATH': '/opt/minemeld/engine/current/bin/mm-restore', 'MINEMELD_LOCAL_PATH': '/opt/minemeld/local', 'MINEMELD_GIT_PATH': '/usr/bin/git', 'SNS_ENABLED': True}
[2019-07-31 15:00:57 UTC] [2266] [INFO] USERS_DB loaded from /opt/minemeld/local/config/api/wsgi.htpasswd
[2019-07-31 15:00:57 UTC] [2266] [INFO] FEEDS_USERS_DB loaded from /opt/minemeld/local/config/api/feeds.htpasswd
[2019-07-31 15:00:57 UTC] [2266] [DEBUG] Instance uuid = 04270110fdf148c090fda4d65771206f
[2019-07-31 15:00:58 UTC] [2266] [INFO] Loading blueprint from taxiing
[2019-07-31 15:00:58 +0000] [2266] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
    worker.init_process()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
    super(GeventWorker, self).init_process()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
    __import__(module)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/main.py", line 3, in <module>
    app = create_app()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 113, in create_app
    app.register_blueprint(bprint)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/flask/app.py", line 64, in wrapper_func
    return f(self, *args, **kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/flask/app.py", line 941, in register_blueprint
    if blueprint.name in self.blueprints:
AttributeError: 'function' object has no attribute 'name'
[2019-07-31 15:00:58 +0000] [2266] [INFO] Worker exiting (pid: 2266)
class Subsystem(tuple):
    'Subsystem(authdb, attrs, enabled, enabled_default)'

    __slots__ = ()

    _fields = ('authdb', 'attrs', 'enabled', 'enabled_default')

    def __new__(_cls, authdb, attrs, enabled, enabled_default):
        'Create new instance of Subsystem(authdb, attrs, enabled, enabled_default)'
        return _tuple.__new__(_cls, (authdb, attrs, enabled, enabled_default))

    @classmethod
    def _make(cls, iterable, new=tuple.__new__, len=len):
        'Make a new Subsystem object from a sequence or iterable'
        result = new(cls, iterable)
        if len(result) != 4:
            raise TypeError('Expected 4 arguments, got %d' % len(result))
        return result

    def __repr__(self):
        'Return a nicely formatted representation string'
        return 'Subsystem(authdb=%r, attrs=%r, enabled=%r, enabled_default=%r)' % self

    def _asdict(self):
        'Return a new OrderedDict which maps field names to their values'
        return OrderedDict(zip(self._fields, self))

    def _replace(_self, **kwds):
        'Return a new Subsystem object replacing specified fields with new values'
        result = _self._make(map(kwds.pop, ('authdb', 'attrs', 'enabled', 'enabled_default'), _self))
        if kwds:
            raise ValueError('Got unexpected field names: %r' % kwds.keys())
        return result

    def __getnewargs__(self):
        'Return self as a plain tuple.  Used by copy and pickle.'
        return tuple(self)

    __dict__ = _property(_asdict)

    def __getstate__(self):
        'Exclude the OrderedDict from pickling'
        pass

    authdb = _property(_itemgetter(0), doc='Alias for field number 0')

    attrs = _property(_itemgetter(1), doc='Alias for field number 1')

    enabled = _property(_itemgetter(2), doc='Alias for field number 2')

    enabled_default = _property(_itemgetter(3), doc='Alias for field number 3')


[2019-07-31 15:00:58 +0000] [2262] [INFO] Shutting down: Master
[2019-07-31 15:00:58 +0000] [2262] [INFO] Reason: Worker failed to boot.

sudo service minemeld restart

 * Restarting: minemeld                                                                                                                                                                        minemeld-supervisord-listener: stopped
minemeld-traced: stopped
minemeld-engine: stopped
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
  return pkg_resources.EntryPoint.parse("x="+spec).load(False)

Re: Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

hshawn — Wed, 31 Jul 2019 18:06:02 GMT

I ended up doing a fresh install of 0.9.62 to resolve the issue for me.