Error Checking credentials - Gateway Timed out

[maltwist]

Hi There,

  I have installed Minemeld on my Ubuntu Server 14.04.. And the service is up and running.. Wheneve I use the default Username and Password to logon to the console, it gives me an error "Error Checking credentials - gateway timed out".. I have also checked the file      opt/minemeld/log/minemeld-web.log for errors ?  but could not find any  particular error code.. 

Can someone help me on this..

Thanks in Advance... 

[lmori]

Hi maltwist,

please could you run the following commands and post the output ?

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

grep ERROR /opt/minemeld/log/minemeld-engine.log  | tail -100

Thanks,

luigi

[maltwist]

Hi lmori,

   Thanks for your reply.. i ran the command and it says "Refused Connection".. 

opt/minemeld/local/supervisor/run/minemeld.sock refused connection

regards

maltwist

[lmori]

It seems that the minemeld service is not running.

Could you check /opt/minemeld/log/minemeld-engine.log and /opt/minemeld/log/supervisord.log  for errors ?

[maltwist]

Checked the Minemeld-engine log and could see  few errors like "Inboundaggregator - Error reading last check point"  

[lmori]

Those errors are normal. Could you try starting the minemeld service (sudo service minemeld start) and then upload the /opt/minemeld/log/supervisord.log  file ?

[maltwist]

Please find the below 

2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-web.conf" during parsing
2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2016-07-29 11:14:53,722 INFO RPC interface 'supervisor' initialized
2016-07-29 11:14:53,722 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-07-29 11:14:53,723 INFO daemonizing the supervisord process
2016-07-29 11:14:53,724 INFO supervisord started with pid 21398
2016-07-29 11:14:54,727 INFO spawned: 'minemeld-engine' with pid 21421
2016-07-29 11:14:54,729 INFO spawned: 'minemeld-traced' with pid 21422
2016-07-29 11:14:54,731 INFO spawned: 'minemeld-web' with pid 21423
2016-07-29 11:14:56,724 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 11:14:56,724 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 11:14:56,724 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:05:35,881 INFO stopped: minemeld-traced (exit status 0)
2016-07-29 12:05:37,826 INFO stopped: minemeld-web (exit status 0)
2016-07-29 12:15:35,931 WARN killing 'minemeld-engine' (21421) with SIGKILL
2016-07-29 12:15:35,951 INFO stopped: minemeld-engine (terminated by SIGKILL)
2016-07-29 12:15:36,039 WARN received SIGTERM indicating exit request
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-web.conf" during parsing
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2016-07-29 12:16:06,274 INFO RPC interface 'supervisor' initialized
2016-07-29 12:16:06,274 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-07-29 12:16:06,275 INFO daemonizing the supervisord process
2016-07-29 12:16:06,276 INFO supervisord started with pid 1208
2016-07-29 12:16:07,279 INFO spawned: 'minemeld-engine' with pid 1322
2016-07-29 12:16:07,282 INFO spawned: 'minemeld-traced' with pid 1323
2016-07-29 12:16:07,285 INFO spawned: 'minemeld-web' with pid 1324
2016-07-29 12:16:08,002 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 12:16:08,660 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:16:08,660 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:16:09,055 INFO spawned: 'minemeld-traced' with pid 1520
2016-07-29 12:16:09,381 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 12:16:11,385 INFO spawned: 'minemeld-traced' with pid 1817
2016-07-29 12:16:12,698 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:23:06,548 INFO exited: minemeld-engine (exit status 0; expected)
2016-07-29 12:23:07,552 INFO spawned: 'minemeld-engine' with pid 2096
2016-07-29 12:23:08,583 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 14:27:31,667 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 14:27:32,671 INFO spawned: 'minemeld-traced' with pid 5451
2016-07-29 14:27:33,962 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

[lmori]

It seems that the trace daemon keeps going down. Could you check /opt/minemeld/log/minemeld-traced.log for errors ?

Thanks.

[maltwist]

Hi,

  sorry for the late response.. here is the log ::

2016-07-29T11:14:54 (21422)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T11:14:54 (21422)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T11:14:54 (21422)main.main INFO: mm-traced config: {}
2016-07-29T11:36:13 (21422)storage.__init__ WARNING: MAX_ID key not found in 00000000579a9c80
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2016-07-29T12:05:35 (21422)writer.stop INFO: Writer - stop called
2016-07-29T12:05:35 (21422)queryprocessor.stop INFO: QueryProcessor - stop called
2016-07-29T12:05:35 (21422)storage.stop INFO: Store - stop called
2016-07-29T12:16:07 (1323)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:07 (1323)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:07 (1323)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 135, in main
comm.start()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 541, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
2016-07-29T12:16:09 (1520)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:09 (1520)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:09 (1520)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 135, in main
comm.start()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 541, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2016-07-29T12:16:11 (1817)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:11 (1817)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:11 (1817)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 516, in _ioloop
self._connections[j].drain_events()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 302, in drain_events
chanmap, None, timeout=timeout,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 365, in _wait_multiple
channel, method_sig, args, content = read_timeout(timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 329, in read_timeout
return self.method_reader.read_method()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
raise m
error: [Errno 110] Connection timed out
<Greenlet at 0x7f279c01a230: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7f279c002410>>(0)> failed with error

2016-07-29T14:27:31 (1817)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 522, in _ioloop_failure
g.get()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
raise self._exception
error: [Errno 110] Connection timed out
2016-07-29T14:27:31 (1817)writer.stop INFO: Writer - stop called
2016-07-29T14:27:31 (1817)queryprocessor.stop INFO: QueryProcessor - stop called
2016-07-29T14:27:31 (1817)storage.stop INFO: Store - stop called
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 150, in main
_cleanup()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 81, in _cleanup
comm.stop()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 598, in stop
rpcc.disconnect()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 301, in disconnect
self.channel.close()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/channel.py", line 176, in close
self._send_method((20, 40), args)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/socket.py", line 460, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/socket.py", line 437, in send
return sock.send(data, flags)
socket.error: [Errno 32] Broken pipe
2016-07-29T14:27:32 (5451)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T14:27:32 (5451)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T14:27:32 (5451)main.main INFO: mm-traced config: {}

[lmori]

I made a mistake in the Ubuntu installation instructions. Could you try the following commands and then restart the instance  ?

sudo ip6tables -A INPUT -i lo -j ACCEPT
sudo ip6tables -P OUTPUT ACCEPT
sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"

I fixed the article as well.

Thanks !

[maltwist]

Hi lmori,

   I tried the above commands and i am still getting the same error as Bad Gateway.. I guess there is something to do with minemeld.sock file in /opt/minemeld/local/supervisor/run    or   minemeld.sock file in /opt/minemeld/local/supervisor/config files.. I am not sure. Looking for your help. 

thanks  

[lmori]

Hi maltwist,

- have you rebooted the instance after applying the changes ?

- could you attach the output of the following commands ?

lsb_release -r -c
sudo netstat -anp
sudo dpkg -l

Thanks,

luigi

[maltwist]

Hi lmori.

  Yes i rebooted.. i ran the above commands and it says all the sream status are connected.. May i know how to start/restart the Minemeld instance. ?

Also when i try to check 

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.

It syas minemeld.sock refused connection.. 

your help will be highly appericiated.. 

regards

Maltwist

[lmori]

Hi MalTwist,

could you attach the output of those commands or send it to me ?

Thanks,

luigi 

[maltwist]

Hi lmori,

  Please find the attached files.

pic 1 - is the reply i got for the  command : 

lsb_release -r -c
sudo netstat -anp
sudo dpkg -l 

 Pic 2 : Is the reply for the command : 

$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord. 

 we need to note that status says : minemeld.sock   refused connection. 

pic 3 :  i suspect some issue on the configuration file and the minemeld.sock  file.. 

thanks