03-21-2019 09:59 AM
First post was marked as spam for some reason so here goes try #2 with a more specific title
Never been able to get a solid ansible deploy to CentOS so stuck with Ubuntu 14 manual install but....after staying up way too late last night trying to figure out why I suddenly could not log into Minemeld, discovering the engine would not start, doing a fresh install on Ubuntu 16.04LTS, confirming all services are started, and then STILL not being able to log in I am now officially asking for help becuase I must be missing something.
I have been running for several months without issues until I tried to log in yesterday. My theory is my Ubuntu 14 instance of Minemeld auto updated and broke something here is the message I get when trying to start it up:
In the UI
ERROR CHECKING CREDENTIALS: INTERNAL SERVER ERROR
In the CLI
# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status
minemeld-engine FATAL Exited too quickly (process log may have details) minemeld-supervisord-listener RUNNING pid 1470, uptime 12:02:55 minemeld-traced RUNNING pid 9459, uptime 1:35:50 minemeld-web RUNNING pid 1473, uptime 12:02:55
# service minemeld start
* Starting: minemeld
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
return pkg_resources.EntryPoint.parse("x="+spec).load(False)
# tail -n 50 /opt/minemeld/log/minemeld-engine.log
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
config=config
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 35, in __init__
super(RedisSet, self).__init__(name, chassis, config)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/actorbase.py", line 17, in __init__
super(ActorBaseFT, self).__init__(*args, **kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 206, in __init__
self.read_checkpoint()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 57, in read_checkpoint
self.last_checkpoint = self.SR.get(self.redis_skey_chkp)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 880, in get
return self.execute_command('GET', name)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
2019-03-21T08:26:53 (10943)launcher.main ERROR: Exception initializing graph
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
mbusmaster.wait_for_chassis(timeout=10)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: checkpoint_graph called, checking current state
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: graph status None, checkpoint_graph ignored
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
mbusmaster.wait_for_chassis(timeout=10)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
will just dump the Ubuntu 14 install forever. I have no idea what the deal is here since all services are running, the install went perfectly fine without any issues, I even changed the admin password from the CLI to make sure I had a password set that I knew. Here is what I am seeing on the Ubuntu 16 install (I used the manual installation method):
In the UI
ERROR CHECKING CREDENTIALS: TIME OUT
In the CLI
# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status
minemeld-engine RUNNING pid 1001, uptime 0:01:40 minemeld-supervisord-listener RUNNING pid 1000, uptime 0:01:40 minemeld-traced RUNNING pid 1002, uptime 0:01:40 minemeld-web RUNNING pid 1005, uptime 0:01:40
# tail -n 50 /opt/minemeld/log/minemeld-supervisor.log
minemeld-supervisord-listener-stdout---supervisor-2zuEqD.log minemeld-traced.log minemeld-traced-stderr---supervisor-dHNP8N.log minemeld-web.log minemeld-web.log.1 minemeld-web-stderr---supervisor-xbdkBx.log supervisord.log root@minemeld:/home/orthonovum# tail -n 50 /opt/minemeld/log/supervisord.log 2019-03-20 19:47:14,019 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:14,019 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:14,019 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:43,833 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs) 2019-03-20 19:51:12,958 WARN received SIGTERM indicating exit request 2019-03-20 19:51:12,958 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die 2019-03-20 19:51:13,443 INFO stopped: minemeld-web (exit status 0) 2019-03-20 19:51:13,467 INFO stopped: minemeld-traced (exit status 0) 2019-03-20 19:51:16,485 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die 2019-03-20 19:51:18,573 INFO stopped: minemeld-engine (exit status 0) 2019-03-20 19:51:18,575 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM) 2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing 2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing 2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing 2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing 2019-03-20 19:51:18,951 INFO RPC interface 'supervisor' initialized 2019-03-20 19:51:18,951 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-03-20 19:51:18,952 INFO daemonizing the supervisord process 2019-03-20 19:51:18,953 INFO supervisord started with pid 1653 2019-03-20 19:51:19,958 INFO spawned: 'minemeld-supervisord-listener' with pid 1654 2019-03-20 19:51:19,960 INFO spawned: 'minemeld-engine' with pid 1655 2019-03-20 19:51:19,963 INFO spawned: 'minemeld-traced' with pid 1656 2019-03-20 19:51:19,971 INFO spawned: 'minemeld-web' with pid 1657 2019-03-20 19:51:20,990 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:20,990 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:20,990 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:50,263 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs) 2019-03-20 20:03:54,120 WARN received SIGTERM indicating exit request 2019-03-20 20:03:54,121 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die 2019-03-20 20:03:54,271 INFO stopped: minemeld-web (exit status 0) 2019-03-20 20:03:54,297 INFO stopped: minemeld-traced (exit status 0) 2019-03-20 20:03:57,315 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die 2019-03-20 20:03:59,400 INFO stopped: minemeld-engine (exit status 0) 2019-03-20 20:03:59,402 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM) 2019-03-21 08:46:35,884 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing 2019-03-21 08:46:35,909 INFO RPC interface 'supervisor' initialized 2019-03-21 08:46:35,909 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-03-21 08:46:35,912 INFO daemonizing the supervisord process 2019-03-21 08:46:35,913 INFO supervisord started with pid 948 2019-03-21 08:46:36,917 INFO spawned: 'minemeld-supervisord-listener' with pid 1000 2019-03-21 08:46:36,920 INFO spawned: 'minemeld-engine' with pid 1001 2019-03-21 08:46:36,924 INFO spawned: 'minemeld-traced' with pid 1002 2019-03-21 08:46:36,936 INFO spawned: 'minemeld-web' with pid 1005 2019-03-21 08:46:38,202 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:46:38,202 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:46:38,202 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:47:07,420 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
Hoping this is something that is easily spotted and fixed (for either Ubuntu 14 or 16) that I am just not knowing what to look for/at. Any ideas here? My EDLs are down atm
TIA
