03-21-2019 09:59 AM
First post was marked as spam for some reason so here goes try #2 with a more specific title
Never been able to get a solid ansible deploy to CentOS so stuck with Ubuntu 14 manual install but....after staying up way too late last night trying to figure out why I suddenly could not log into Minemeld, discovering the engine would not start, doing a fresh install on Ubuntu 16.04LTS, confirming all services are started, and then STILL not being able to log in I am now officially asking for help becuase I must be missing something.
I have been running for several months without issues until I tried to log in yesterday. My theory is my Ubuntu 14 instance of Minemeld auto updated and broke something here is the message I get when trying to start it up:
In the UI
ERROR CHECKING CREDENTIALS: INTERNAL SERVER ERROR
In the CLI
# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status
minemeld-engine FATAL Exited too quickly (process log may have details) minemeld-supervisord-listener RUNNING pid 1470, uptime 12:02:55 minemeld-traced RUNNING pid 9459, uptime 1:35:50 minemeld-web RUNNING pid 1473, uptime 12:02:55
# service minemeld start
* Starting: minemeld
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
return pkg_resources.EntryPoint.parse("x="+spec).load(False)
# tail -n 50 /opt/minemeld/log/minemeld-engine.log
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
config=config
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 35, in __init__
super(RedisSet, self).__init__(name, chassis, config)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/actorbase.py", line 17, in __init__
super(ActorBaseFT, self).__init__(*args, **kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 206, in __init__
self.read_checkpoint()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/ft/redis.py", line 57, in read_checkpoint
self.last_checkpoint = self.SR.get(self.redis_skey_chkp)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 880, in get
return self.execute_command('GET', name)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/client.py", line 578, in execute_command
connection.send_command(*args)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 563, in send_command
self.send_packed_command(self.pack_command(*args))
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
2019-03-21T08:26:53 (10943)launcher.main ERROR: Exception initializing graph
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
mbusmaster.wait_for_chassis(timeout=10)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: checkpoint_graph called, checking current state
2019-03-21T08:26:53 (10943)mgmtbus.checkpoint_graph INFO: graph status None, checkpoint_graph ignored
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 284, in main
mbusmaster.wait_for_chassis(timeout=10)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/mgmtbus.py", line 141, in wait_for_chassis
raise RuntimeError('Timeout waiting for chassis')
RuntimeError: Timeout waiting for chassis
will just dump the Ubuntu 14 install forever. I have no idea what the deal is here since all services are running, the install went perfectly fine without any issues, I even changed the admin password from the CLI to make sure I had a password set that I knew. Here is what I am seeing on the Ubuntu 16 install (I used the manual installation method):
In the UI
ERROR CHECKING CREDENTIALS: TIME OUT
In the CLI
# sudo -u minemeld /opt/minemeld/engine/current/bin/supervisor /opt/minemeld/supervisor/config/supervisord.conf status
minemeld-engine RUNNING pid 1001, uptime 0:01:40 minemeld-supervisord-listener RUNNING pid 1000, uptime 0:01:40 minemeld-traced RUNNING pid 1002, uptime 0:01:40 minemeld-web RUNNING pid 1005, uptime 0:01:40
# tail -n 50 /opt/minemeld/log/minemeld-supervisor.log
minemeld-supervisord-listener-stdout---supervisor-2zuEqD.log minemeld-traced.log minemeld-traced-stderr---supervisor-dHNP8N.log minemeld-web.log minemeld-web.log.1 minemeld-web-stderr---supervisor-xbdkBx.log supervisord.log root@minemeld:/home/orthonovum# tail -n 50 /opt/minemeld/log/supervisord.log 2019-03-20 19:47:14,019 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:14,019 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:14,019 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:47:43,833 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs) 2019-03-20 19:51:12,958 WARN received SIGTERM indicating exit request 2019-03-20 19:51:12,958 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die 2019-03-20 19:51:13,443 INFO stopped: minemeld-web (exit status 0) 2019-03-20 19:51:13,467 INFO stopped: minemeld-traced (exit status 0) 2019-03-20 19:51:16,485 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die 2019-03-20 19:51:18,573 INFO stopped: minemeld-engine (exit status 0) 2019-03-20 19:51:18,575 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM) 2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing 2019-03-20 19:51:18,938 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing 2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing 2019-03-20 19:51:18,939 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing 2019-03-20 19:51:18,951 INFO RPC interface 'supervisor' initialized 2019-03-20 19:51:18,951 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-03-20 19:51:18,952 INFO daemonizing the supervisord process 2019-03-20 19:51:18,953 INFO supervisord started with pid 1653 2019-03-20 19:51:19,958 INFO spawned: 'minemeld-supervisord-listener' with pid 1654 2019-03-20 19:51:19,960 INFO spawned: 'minemeld-engine' with pid 1655 2019-03-20 19:51:19,963 INFO spawned: 'minemeld-traced' with pid 1656 2019-03-20 19:51:19,971 INFO spawned: 'minemeld-web' with pid 1657 2019-03-20 19:51:20,990 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:20,990 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:20,990 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-20 19:51:50,263 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs) 2019-03-20 20:03:54,120 WARN received SIGTERM indicating exit request 2019-03-20 20:03:54,121 INFO waiting for minemeld-engine, minemeld-traced, minemeld-web, minemeld-supervisord-listener to die 2019-03-20 20:03:54,271 INFO stopped: minemeld-web (exit status 0) 2019-03-20 20:03:54,297 INFO stopped: minemeld-traced (exit status 0) 2019-03-20 20:03:57,315 INFO waiting for minemeld-engine, minemeld-supervisord-listener to die 2019-03-20 20:03:59,400 INFO stopped: minemeld-engine (exit status 0) 2019-03-20 20:03:59,402 INFO stopped: minemeld-supervisord-listener (terminated by SIGTERM) 2019-03-21 08:46:35,884 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing 2019-03-21 08:46:35,885 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing 2019-03-21 08:46:35,909 INFO RPC interface 'supervisor' initialized 2019-03-21 08:46:35,909 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-03-21 08:46:35,912 INFO daemonizing the supervisord process 2019-03-21 08:46:35,913 INFO supervisord started with pid 948 2019-03-21 08:46:36,917 INFO spawned: 'minemeld-supervisord-listener' with pid 1000 2019-03-21 08:46:36,920 INFO spawned: 'minemeld-engine' with pid 1001 2019-03-21 08:46:36,924 INFO spawned: 'minemeld-traced' with pid 1002 2019-03-21 08:46:36,936 INFO spawned: 'minemeld-web' with pid 1005 2019-03-21 08:46:38,202 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:46:38,202 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:46:38,202 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-03-21 08:47:07,420 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
Hoping this is something that is easily spotted and fixed (for either Ubuntu 14 or 16) that I am just not knowing what to look for/at. Any ideas here? My EDLs are down atm
TIA
03-22-2019 08:32 AM - edited 03-22-2019 11:03 AM
Update to this. I tried the Ubuntu 16.04LTS Ansible install too. The install goes off without a hitch but I get the same issue/error/unable to log in after installing and rebooting. Just because I like to torture myself I am going to also try the CentOS install (Which I have working in a production environment but it is an older version of Minemeld) and see if I get any luck there.
I just want a working Minemeld. But I am really having to work at it 😄
UPDATE: Same thing with fresh CentOS7 ansible install. Beginning to think there is a bug somewhere... (>.>)
UPDATE#2: Looks like python2.7 is using udp/5000 and the minemeld web log is constantly complaining about the inability to connect to 127.0.0.1:5000. If I kill the python process using 5000 then I can log in, but of course its all messed up and says "ERROR RETRIEVING OUTPUT METRIX: Timeout"
So something python is running on udp port 5000 is interferring with the ability to log into the web UI but whatever this is must be a critical process so killing it is not the answer. I am past my expertise with minemeld so hopefully someone intimately familiar with the way it is designed can help out.
web log spams the following
[ERROR] Can't connect to ('127.0.0.1', 5000)
UPDATE #3: Fresh install of Ubuntu 14LTS then fresh manual install of Minemeld = same issue unable to log in. 99.99% sure at this point that it is nothing I am doing or my VMs but something in the minemeld code.
04-23-2019 01:25 PM
Hi mate,
I was experiencing the same problem. It appears that I have got arround this by killing the processes on 5000. I haven't come across any other problems after doing this.
I ran the following command:
sudo fuser -k 5000/tcp
Hope this helps, let me know how you get on...
Ryan
07-31-2019 08:33 AM
Im having the same issues as @hshawn. This was working for several weeks and now I cant log in. I get the: ERROR CHECKING CREDENTIALS: Bad Gateway error message. Here is my output.
lsb_release -r
Release: 14.04
sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine RUNNING pid 1943, uptime 0:17:12 minemeld-supervisord-listener RUNNING pid 1942, uptime 0:17:12 minemeld-traced RUNNING pid 1944, uptime 0:17:12 minemeld-web FATAL Exited too quickly (process log may have details)
tail -n 100 /opt/minemeld/log/minemeld-web.log
[2019-07-31 15:00:54 +0000] [2252] [INFO] Shutting down: Master
[2019-07-31 15:00:54 +0000] [2252] [INFO] Reason: Worker failed to boot.
[2019-07-31 15:00:57 +0000] [2262] [INFO] Starting gunicorn 19.5.0
[2019-07-31 15:00:57 +0000] [2262] [INFO] Listening at: http://127.0.0.1:5000 (2262)
[2019-07-31 15:00:57 +0000] [2262] [INFO] Using worker: gevent
[2019-07-31 15:00:57 +0000] [2266] [INFO] Booting worker with pid: 2266
[2019-07-31 15:00:57 UTC] [2266] [INFO] MMLogger started
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/aaa.py:21: ExtDeprecationWarning: Importing flask.ext.login is deprecated, use flask_login instead.
import flask.ext.login
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.DeprecatedIn23,
[2019-07-31 15:00:57 UTC] [2266] [INFO] Config loaded: {'MINEMELD_LOCAL_LIBRARY_PATH': '/opt/minemeld/local/library', 'USERS_DB': 'wsgi.htpasswd', 'MINEMELD_LOCAL_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes', 'SUPERVISOR_URL': 'unix:///var/run/minemeld/minemeld.sock', 'MINEMELD_PROTOTYPE_PATH': '/opt/minemeld/local/prototypes/:/opt/minemeld/prototypes/current', 'MINEMELD_PIP_PATH': '/opt/minemeld/engine/current/bin/pip', 'MINEMELD_TRACED_PURGE_PATH': '/opt/minemeld/engine/current/bin/mm-traced-purge', 'API_AUTH_ENABLED': True, 'MINEMELD_LOG_DIRECTORY_PATH': '/opt/minemeld/log', 'MINEMELD_7Z_PATH': '/usr/bin/7z', 'MINEMELD_RESTORE_PATH': '/opt/minemeld/engine/current/bin/mm-restore', 'MINEMELD_LOCAL_PATH': '/opt/minemeld/local', 'MINEMELD_GIT_PATH': '/usr/bin/git', 'SNS_ENABLED': True}
[2019-07-31 15:00:57 UTC] [2266] [INFO] USERS_DB loaded from /opt/minemeld/local/config/api/wsgi.htpasswd
[2019-07-31 15:00:57 UTC] [2266] [INFO] FEEDS_USERS_DB loaded from /opt/minemeld/local/config/api/feeds.htpasswd
[2019-07-31 15:00:57 UTC] [2266] [DEBUG] Instance uuid = 04270110fdf148c090fda4d65771206f
[2019-07-31 15:00:58 UTC] [2266] [INFO] Loading blueprint from taxiing
[2019-07-31 15:00:58 +0000] [2266] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
worker.init_process()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
super(GeventWorker, self).init_process()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/main.py", line 3, in <module>
app = create_app()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/flask/__init__.py", line 113, in create_app
app.register_blueprint(bprint)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/flask/app.py", line 64, in wrapper_func
return f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/flask/app.py", line 941, in register_blueprint
if blueprint.name in self.blueprints:
AttributeError: 'function' object has no attribute 'name'
[2019-07-31 15:00:58 +0000] [2266] [INFO] Worker exiting (pid: 2266)
class Subsystem(tuple):
'Subsystem(authdb, attrs, enabled, enabled_default)'
__slots__ = ()
_fields = ('authdb', 'attrs', 'enabled', 'enabled_default')
def __new__(_cls, authdb, attrs, enabled, enabled_default):
'Create new instance of Subsystem(authdb, attrs, enabled, enabled_default)'
return _tuple.__new__(_cls, (authdb, attrs, enabled, enabled_default))
@classmethod
def _make(cls, iterable, new=tuple.__new__, len=len):
'Make a new Subsystem object from a sequence or iterable'
result = new(cls, iterable)
if len(result) != 4:
raise TypeError('Expected 4 arguments, got %d' % len(result))
return result
def __repr__(self):
'Return a nicely formatted representation string'
return 'Subsystem(authdb=%r, attrs=%r, enabled=%r, enabled_default=%r)' % self
def _asdict(self):
'Return a new OrderedDict which maps field names to their values'
return OrderedDict(zip(self._fields, self))
def _replace(_self, **kwds):
'Return a new Subsystem object replacing specified fields with new values'
result = _self._make(map(kwds.pop, ('authdb', 'attrs', 'enabled', 'enabled_default'), _self))
if kwds:
raise ValueError('Got unexpected field names: %r' % kwds.keys())
return result
def __getnewargs__(self):
'Return self as a plain tuple. Used by copy and pickle.'
return tuple(self)
__dict__ = _property(_asdict)
def __getstate__(self):
'Exclude the OrderedDict from pickling'
pass
authdb = _property(_itemgetter(0), doc='Alias for field number 0')
attrs = _property(_itemgetter(1), doc='Alias for field number 1')
enabled = _property(_itemgetter(2), doc='Alias for field number 2')
enabled_default = _property(_itemgetter(3), doc='Alias for field number 3')
[2019-07-31 15:00:58 +0000] [2262] [INFO] Shutting down: Master
[2019-07-31 15:00:58 +0000] [2262] [INFO] Reason: Worker failed to boot.
sudo service minemeld restart
* Restarting: minemeld minemeld-supervisord-listener: stopped
minemeld-traced: stopped
minemeld-engine: stopped
/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
return pkg_resources.EntryPoint.parse("x="+spec).load(False)
07-31-2019 11:06 AM
I ended up doing a fresh install of 0.9.62 to resolve the issue for me.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
