Minemeld Crashing

[pwebber]

Hello,

I'm have recently started to try to utilize a few different feeds and as such have been trying to get Minemeld to process the feeds. I'm using the MISP extensions as well as CIF and TAXII prototypes to pull in these feeds. I noticed though every morning when I check on Minemeld it has crashed. By this I mean that:

- the green loading bar keeps running across the screen

- the nodes page won't load

- TAXII output prototype is giving a bad gateway 502 to my TAXII clients.

If I reboot the server it works for a while, then crashes again. Not sure if this helps but I'm posting some of the errors I see in the engine log:

Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 561, in _ioloop
    conn.drain_events()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
    return amqp_method(channel, args)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
    reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'torexitnodes' in vhost '/'
<Greenlet at 0x7fa1d1fa1af0: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7fa1d9ae4d10>>(12)> failed with NotFound

2019-09-24T16:44:47 (1620)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 567, in _ioloop_failure
    g.get()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
    raise self._exception
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'torexitnodes' in vhost '/'
2019-09-24T16:44:47 (1620)chassis.stop INFO: chassis stop called

2019-09-24T15:59:29 (1623)client.search INFO: trying to decompress...
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 561, in _ioloop
    conn.drain_events()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
    return amqp_method(channel, args)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
    reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'REN-ISAC-SES-MINER-DOMAIN' in vhost '/'
<Greenlet at 0x7fa1d5f41e10: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7fa1d9ae40d0>>(13)> failed with NotFound

2019-09-24T15:59:29 (1623)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 567, in _ioloop_failure
    g.get()
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
    raise self._exception
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'SOME NAME' in vhost '/'

 There are also errors processing items in some fo the feeds as well, but they don't seem to be a hard failures, I mean it looks like Minemeld just ignores them. There is a lot of this as well after it crashes:

2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _id: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:30 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:30 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False

 Also seeing this in the minemeld-traced.log:

Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
    self.handler(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
    raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2019-09-24T13:56:47 (11818)writer.stop INFO: Writer - stop called
2019-09-24T13:56:47 (11818)queryprocessor.stop INFO: QueryProcessor - stop called
2019-09-24T13:56:47 (11818)storage.stop INFO: Store - stop called
2019-09-24T13:59:15 (1597)main.main INFO: Starting mm-traced version 0.9.52.post1
2019-09-24T13:59:15 (1597)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2019-09-24T13:59:15 (1597)main.main INFO: mm-traced config: {}
2019-09-24T13:59:26 (1597)storage.__init__ WARNING: MAX_ID key not found in 000000005d895c80
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
    self.handler(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
    raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2019-09-25T06:36:07 (1597)writer.stop INFO: Writer - stop called
2019-09-25T06:36:07 (1597)queryprocessor.stop INFO: QueryProcessor - stop called
2019-09-25T06:36:07 (1597)storage.stop INFO: Store - stop called
2019-09-25T06:36:13 (14365)main.main INFO: Starting mm-traced version 0.9.52.post1
2019-09-25T06:36:13 (14365)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2019-09-25T06:36:13 (14365)main.main INFO: mm-traced config: {}
root@minemeld:/opt/minemeld/log# cat minemeld-traced.log