Miners do not collect any Information when more than 128 nodes

[Remo]

Hello,

I have a problem with my minemeld server. Some days ago I reached 128 nodes (sum of miners, processors and outputs). If I do a commit with 128 nodes or more the minemeld process seems to keep restarting over and over. As soon as I reduce to 127 nodes or less it works again.

In the logs I did not find a lot ...

2018-07-24T16:18:32 (7301)queryprocessor.__init__ INFO: Query 322ca06b-e087-43db-8970-c52bf86efecc -
2018-07-24T16:18:32 (7301)queryprocessor._core_run INFO: Query 322ca06b-e087-43db-8970-c52bf86efecc finished - 100
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4be000
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b551a80
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4d3180
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4fd480
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
    self.handler(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 75, in _sigterm_handler
    raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2018-07-24T16:21:01 (7301)writer.stop INFO: Writer - stop called
2018-07-24T16:21:01 (7301)queryprocessor.stop INFO: QueryProcessor - stop called
2018-07-24T16:21:01 (7301)storage.stop INFO: Store - stop called
2018-07-24T16:27:43 (962)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:43 (962)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:43 (962)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
2018-07-24T16:27:45 (1939)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:45 (1939)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:45 (1939)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2018-07-24T16:27:47 (3575)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:47 (3575)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:47 (3575)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2018-07-24T16:27:50 (4843)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:50 (4843)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:50 (4843)main.main INFO: mm-traced config: {}

Is this an existing bug or is there at least a solution on how to get more than 128 nodes?

Thanks in advance for your help.

Remo

[danilo.souza]

Hi @Remo

I already had this problem twice, but with two diagnosis. In both cases my MM kept restarting over and over. In the first case I added more memory to my VM and it stopped. But in the second case, my MM was not able to deduplicate IPs from many lists in the same Aggregator. Look the discussion here:

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Output-limit/td-p/210302

You could try both of them and report here the results.

Best regards.