Bad Gateway Error - Minemeld Not Running

[AaronFaby]

Hi All,

My Minemeld instance seemed to randomly break and I'm not sure why. When I try to login I get a bad gateway error and the EDL URL's give the same message. Here are some log snippets:

minemeld-engine.log:

2018-05-11T06:25:45 (4222)base.state INFO: bruteforce_de-IP - transitioning to state 8
2018-05-11T06:25:45 (4222)basepoller.stop INFO: bruteforce_de-IP - # indicators: 350
2018-05-11T06:25:45 (4222)base.state INFO: office365_officeOnline - transitioning to state 8
2018-05-11T06:25:45 (4222)basepoller.stop INFO: office365_officeOnline - # indicators: 93
2018-05-11T06:25:45 (4222)base.state INFO: inboundfeedmc - transitioning to state 8
2018-05-11T06:25:45 (4222)chassis.stop INFO: Stopping fabric
2018-05-11T06:25:45 (4222)chassis.stop INFO: Stopping mgmtbus
2018-05-11T06:25:45 (4222)chassis.stop INFO: chassis - stopped
2018-05-11T06:25:45 (4222)launcher._run_chassis INFO: power off
2018-05-11T06:25:45 (4214)launcher.main INFO: One of the chassis has stopped, exit

minemeld-web.log:

[2018-05-11 06:25:41 +0000] [12158] [INFO] Worker exiting (pid: 12158)
class Subsystem(tuple):
'Subsystem(authdb, attrs, enabled, enabled_default)'

__slots__ = ()

_fields = ('authdb', 'attrs', 'enabled', 'enabled_default')

def __new__(_cls, authdb, attrs, enabled, enabled_default):
'Create new instance of Subsystem(authdb, attrs, enabled, enabled_default)'
return _tuple.__new__(_cls, (authdb, attrs, enabled, enabled_default))

@classmethod
def _make(cls, iterable, new=tuple.__new__, len=len):
'Make a new Subsystem object from a sequence or iterable'
result = new(cls, iterable)
if len(result) != 4:
raise TypeError('Expected 4 arguments, got %d' % len(result))
return result

def __repr__(self):
'Return a nicely formatted representation string'
return 'Subsystem(authdb=%r, attrs=%r, enabled=%r, enabled_default=%r)' % self

def _asdict(self):
'Return a new OrderedDict which maps field names to their values'
return OrderedDict(zip(self._fields, self))

def _replace(_self, **kwds):
'Return a new Subsystem object replacing specified fields with new values'
result = _self._make(map(kwds.pop, ('authdb', 'attrs', 'enabled', 'enabled_default'), _self))
if kwds:
raise ValueError('Got unexpected field names: %r' % kwds.keys())
return result

def __getnewargs__(self):
'Return self as a plain tuple. Used by copy and pickle.'
return tuple(self)

__dict__ = _property(_asdict)

def __getstate__(self):
'Exclude the OrderedDict from pickling'
pass

authdb = _property(_itemgetter(0), doc='Alias for field number 0')

attrs = _property(_itemgetter(1), doc='Alias for field number 1')

enabled = _property(_itemgetter(2), doc='Alias for field number 2')

enabled_default = _property(_itemgetter(3), doc='Alias for field number 3')

[2018-05-11 06:25:42 +0000] [12129] [INFO] Shutting down: Master

minemeld-trace.log:

2018-05-10T00:20:52 (12926)storage.__init__ WARNING: MAX_ID key not found in 000000005af38b80
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2018-05-10T06:25:46 (12926)writer.stop INFO: Writer - stop called
2018-05-10T06:25:46 (12926)queryprocessor.stop INFO: QueryProcessor - stop called
2018-05-10T06:25:46 (12926)storage.stop INFO: Store - stop called
2018-05-10T06:25:52 (8632)main.main INFO: Starting mm-traced version 0.9.44
2018-05-10T06:25:52 (8632)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-05-10T06:25:52 (8632)main.main INFO: mm-traced config: {}
2018-05-11T00:02:27 (8632)storage.__init__ WARNING: MAX_ID key not found in 000000005af4dd00
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.44/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2018-05-11T06:25:41 (8632)writer.stop INFO: Writer - stop called
2018-05-11T06:25:41 (8632)queryprocessor.stop INFO: QueryProcessor - stop called
2018-05-11T06:25:41 (8632)storage.stop INFO: Store - stop called

Please advise. Thanks!

[xhoms]

Hi @AaronFaby,

looks like MineMeld services are not running.

You can check them with the command

$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

[AaronFaby]

I tried that but the command was not missing. I had to rollback to the previous version because the current version was missing a ton of files.