Miners do not collect any Information when more than 128 nodes

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Miners do not collect any Information when more than 128 nodes

L7 Applicator

Hello,

 

I have a problem with my minemeld server. Some days ago I reached 128 nodes (sum of miners, processors and outputs). If I do a commit with 128 nodes or more the minemeld process seems to keep restarting over and over. As soon as I reduce to 127 nodes or less it works again.

In the logs I did not find a lot ...

2018-07-24T16:18:32 (7301)queryprocessor.__init__ INFO: Query 322ca06b-e087-43db-8970-c52bf86efecc -
2018-07-24T16:18:32 (7301)queryprocessor._core_run INFO: Query 322ca06b-e087-43db-8970-c52bf86efecc finished - 100
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4be000
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b551a80
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4d3180
2018-07-24T16:18:32 (7301)storage.remove_reference WARNING: Attempt to remove non existing reference: 322ca06b-e087-43db-8970-c52bf86efecc - 000000005b4fd480
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
    self.handler(*self.args, **self.kwargs)
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 75, in _sigterm_handler
    raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2018-07-24T16:21:01 (7301)writer.stop INFO: Writer - stop called
2018-07-24T16:21:01 (7301)queryprocessor.stop INFO: QueryProcessor - stop called
2018-07-24T16:21:01 (7301)storage.stop INFO: Store - stop called
2018-07-24T16:27:43 (962)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:43 (962)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:43 (962)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
2018-07-24T16:27:45 (1939)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:45 (1939)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:45 (1939)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2018-07-24T16:27:47 (3575)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:47 (3575)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:47 (3575)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
  File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
    load_entry_point('minemeld-core', 'console_scripts', 'mm-traced')()
  File "/opt/minemeld/engine/core/minemeld/traced/main.py", line 136, in main
    comm.start()
  File "/opt/minemeld/engine/core/minemeld/comm/amqp.py", line 595, in start
    c = amqp.connection.Connection(**self.amqp_config)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/opt/minemeld/engine/current/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2018-07-24T16:27:50 (4843)main.main INFO: Starting mm-traced version 0.9.44.post1
2018-07-24T16:27:50 (4843)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced/traced.yml', verbose=False)
2018-07-24T16:27:50 (4843)main.main INFO: mm-traced config: {}

Is this an existing bug or is there at least a solution on how to get more than 128 nodes?

 

Thanks in advance for your help.

Remo

1 REPLY 1

L4 Transporter

Hi @Remo

 

I already had this problem twice, but with two diagnosis. In both cases my MM kept restarting over and over. In the first case I added more memory to my VM and it stopped. But in the second case, my MM was not able to deduplicate IPs from many lists in the same Aggregator. Look the discussion here:

 

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Output-limit/td-p/210302

 

You could try both of them and report here the results.

 

Best regards.

 

  • 2535 Views
  • 1 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!