Who Me Too'd this topic

Who Me Too'd this topic

L4 Transporter

Sudden login issues with Minemeld on Ubuntu 14 and can't login under Ubuntu 16

First post was marked as spam for some reason so here goes try #2 with a more specific title

 

Never been able to get a solid ansible deploy to CentOS so stuck with Ubuntu 14 manual install but....after staying up way too late last night trying to figure out why I suddenly could not log into Minemeld, discovering the engine would not start, doing a fresh install on Ubuntu 16.04LTS, confirming all services are started, and then STILL not being able to log in I am now officially asking for help becuase I must be missing something.

 

I have been running for several months without issues until I tried to log in yesterday. My theory is my Ubuntu 14 instance of Minemeld auto updated and broke something here is the message I get when trying to start it up:

 

In the UI

ERROR CHECKING CREDENTIALS: INTERNAL SERVER ERROR

 

In the CLI

# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor  /opt/minemeld/supervisor/config/supervisord.conf status

minemeld-engine                  FATAL     Exited too quickly (process log may have details)
minemeld-supervisord-listener    RUNNING   pid 1470, uptime 12:02:55
minemeld-traced                  RUNNING   pid 9459, uptime 1:35:50
minemeld-web                     RUNNING   pid 1473, uptime 12:02:55

# service minemeld start

 * Starting: minemeld                                                                                               
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
  return pkg_resources.EntryPoint.parse("x="+spec).load(False)

# tail -n 50 /opt/minemeld/log/minemeld-engine.log

 raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
    config=config
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 35, in __init__
    super(RedisSet, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/actorbase.py", line 17, in __init__
    super(ActorBaseFT, self).__init__(*args, **kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 206, in __init__
    self.read_checkpoint()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 57, in read_checkpoint
    self.last_checkpoint = self.SR.get(self.redis_skey_chkp)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 880, in get
    return self.execute_command('GET', name)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
    raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
2019-03-21T08:26:53 (10943)launcher.main ERROR: Exception initializing graph
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
    mbusmaster.wait_for_chassis(timeout=10)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
    raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: checkpoint_graph called, checking current state
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: graph status None, checkpoint_graph ignored
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
    sys.exit(main())
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
    mbusmaster.wait_for_chassis(timeout=10)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
    raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis

 

will just dump the Ubuntu 14 install forever.  I have no idea what the deal is here since all services are running, the install went perfectly fine without any issues, I even changed the admin password from the CLI to make sure I had a password set that I knew. Here is what I am seeing on the Ubuntu 16 install (I used the manual installation method):

 

In the UI

ERROR CHECKING CREDENTIALS: TIME OUT

 

In the CLI

# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor  /opt/minemeld/supervisor/config/supervisord.conf status

minemeld-engine                  RUNNING   pid 1001, uptime 0:01:40
minemeld-supervisord-listener    RUNNING   pid 1000, uptime 0:01:40
minemeld-traced                  RUNNING   pid 1002, uptime 0:01:40
minemeld-web                     RUNNING   pid 1005, uptime 0:01:40

# tail -n 50 /opt/minemeld/log/minemeld-supervisor.log

minemeld-supervisord-listener-stdout---supervisor-2zuEqD.log
minemeld-traced.log
minemeld-traced-stderr---supervisor-dHNP8N.log
minemeld-web.log
minemeld-web.log.1
minemeld-web-stderr---supervisor-xbdkBx.log
supervisord.log
root@minemeld:/home/orthonovum# tail -n 50 /opt/minemeld/log/supervisord.log
2019-03-20 19:47:14,019 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:14,019 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:14,019 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:47:43,833 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2019-03-20 19:51:12,958 WARN received SIGTERM indicating exit request
2019-03-20 19:51:12,958 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die
2019-03-20 19:51:13,443 INFO stopped: minemeld-web (exit status 0)
2019-03-20 19:51:13,467 INFO stopped: minemeld-traced (exit status 0)
2019-03-20 19:51:16,485 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die
2019-03-20 19:51:18,573 INFO stopped: minemeld-engine (exit status 0)
2019-03-20 19:51:18,575 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM)
2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing
2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing
2019-03-20 19:51:18,951 INFO RPC interface 'supervisor' initialized
2019-03-20 19:51:18,951 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-03-20 19:51:18,952 INFO daemonizing the supervisord process
2019-03-20 19:51:18,953 INFO supervisord started with pid 1653
2019-03-20 19:51:19,958 INFO spawned: 'minemeld-supervisord-listener' with pid 1654
2019-03-20 19:51:19,960 INFO spawned: 'minemeld-engine' with pid 1655
2019-03-20 19:51:19,963 INFO spawned: 'minemeld-traced' with pid 1656
2019-03-20 19:51:19,971 INFO spawned: 'minemeld-web' with pid 1657
2019-03-20 19:51:20,990 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:20,990 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:20,990 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-20 19:51:50,263 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2019-03-20 20:03:54,120 WARN received SIGTERM indicating exit request
2019-03-20 20:03:54,121 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die
2019-03-20 20:03:54,271 INFO stopped: minemeld-web (exit status 0)
2019-03-20 20:03:54,297 INFO stopped: minemeld-traced (exit status 0)
2019-03-20 20:03:57,315 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die
2019-03-20 20:03:59,400 INFO stopped: minemeld-engine (exit status 0)
2019-03-20 20:03:59,402 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM)
2019-03-21 08:46:35,884 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing
2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing
2019-03-21 08:46:35,909 INFO RPC interface 'supervisor' initialized
2019-03-21 08:46:35,909 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-03-21 08:46:35,912 INFO daemonizing the supervisord process
2019-03-21 08:46:35,913 INFO supervisord started with pid 948
2019-03-21 08:46:36,917 INFO spawned: 'minemeld-supervisord-listener' with pid 1000
2019-03-21 08:46:36,920 INFO spawned: 'minemeld-engine' with pid 1001
2019-03-21 08:46:36,924 INFO spawned: 'minemeld-traced' with pid 1002
2019-03-21 08:46:36,936 INFO spawned: 'minemeld-web' with pid 1005
2019-03-21 08:46:38,202 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:46:38,202 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:46:38,202 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-21 08:47:07,420 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)

Hoping this is something that is easily spotted and fixed (for either Ubuntu 14 or 16) that I am just not knowing what to look for/at. Any ideas here? My EDLs are down atm

 

TIA

 

 

Who Me Too'd this topic