- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
09-25-2019 07:39 AM - edited 09-25-2019 07:40 AM
Hello,
I'm have recently started to try to utilize a few different feeds and as such have been trying to get Minemeld to process the feeds. I'm using the MISP extensions as well as CIF and TAXII prototypes to pull in these feeds. I noticed though every morning when I check on Minemeld it has crashed. By this I mean that:
- the green loading bar keeps running across the screen
- the nodes page won't load
- TAXII output prototype is giving a bad gateway 502 to my TAXII clients.
If I reboot the server it works for a while, then crashes again. Not sure if this helps but I'm posting some of the errors I see in the engine log:
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 561, in _ioloop
conn.drain_events()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
return amqp_method(channel, args)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'torexitnodes' in vhost '/'
<Greenlet at 0x7fa1d1fa1af0: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7fa1d9ae4d10>>(12)> failed with NotFound
2019-09-24T16:44:47 (1620)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 567, in _ioloop_failure
g.get()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
raise self._exception
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'torexitnodes' in vhost '/'
2019-09-24T16:44:47 (1620)chassis.stop INFO: chassis stop called
2019-09-24T15:59:29 (1623)client.search INFO: trying to decompress...
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 561, in _ioloop
conn.drain_events()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/connection.py", line 323, in drain_events
return amqp_method(channel, args)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'REN-ISAC-SES-MINER-DOMAIN' in vhost '/'
<Greenlet at 0x7fa1d5f41e10: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7fa1d9ae40d0>>(13)> failed with NotFound
2019-09-24T15:59:29 (1623)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 567, in _ioloop_failure
g.get()
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
raise self._exception
NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'SOME NAME' in vhost '/'
There are also errors processing items in some fo the feeds as well, but they don't seem to be a hard failures, I mean it looks like Minemeld just ignores them. There is a lot of this as well after it crashes:
2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T06:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T07:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:25 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:26 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:27 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _id: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T08:59:29 (1623)table._query_by_index INFO: Deleted in scan of _last_run: 0
2019-09-25T08:59:30 (1623)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-09-25T08:59:30 (1623)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T09:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T10:59:26 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T11:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T12:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
2019-09-25T13:59:27 (1623)basepoller._huppable_wait INFO: hup is clear: False
Also seeing this in the minemeld-traced.log:
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2019-09-24T13:56:47 (11818)writer.stop INFO: Writer - stop called
2019-09-24T13:56:47 (11818)queryprocessor.stop INFO: QueryProcessor - stop called
2019-09-24T13:56:47 (11818)storage.stop INFO: Store - stop called
2019-09-24T13:59:15 (1597)main.main INFO: Starting mm-traced version 0.9.52.post1
2019-09-24T13:59:15 (1597)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2019-09-24T13:59:15 (1597)main.main INFO: mm-traced config: {}
2019-09-24T13:59:26 (1597)storage.__init__ WARNING: MAX_ID key not found in 000000005d895c80
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.52.post1/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2019-09-25T06:36:07 (1597)writer.stop INFO: Writer - stop called
2019-09-25T06:36:07 (1597)queryprocessor.stop INFO: QueryProcessor - stop called
2019-09-25T06:36:07 (1597)storage.stop INFO: Store - stop called
2019-09-25T06:36:13 (14365)main.main INFO: Starting mm-traced version 0.9.52.post1
2019-09-25T06:36:13 (14365)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2019-09-25T06:36:13 (14365)main.main INFO: mm-traced config: {}
root@minemeld:/opt/minemeld/log# cat minemeld-traced.log
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!