Error Checking credentials - Gateway Timed out

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Error Checking credentials - Gateway Timed out

L2 Linker

Hi There,

  I have installed Minemeld on my Ubuntu Server 14.04.. And the service is up and running.. Wheneve I use the default Username and Password to logon to the console, it gives me an error "Error Checking credentials - gateway timed out".. I have also checked the file      opt/minemeld/log/minemeld-web.log for errors ?  but could not find any  particular error code.. 

 

Can someone help me on this..

Thanks in Advance... 

15 REPLIES 15

L7 Applicator

Hi maltwist,

please could you run the following commands and post the output ?

 

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

grep ERROR /opt/minemeld/log/minemeld-engine.log  | tail -100

Thanks,

luigi

Hi lmori,

   Thanks for your reply.. i ran the command and it says "Refused Connection".. 

opt/minemeld/local/supervisor/run/minemeld.sock refused connection

regards

maltwist

It seems that the minemeld service is not running.

 

Could you check /opt/minemeld/log/minemeld-engine.log and /opt/minemeld/log/supervisord.log  for errors ?

Checked the Minemeld-engine log and could see  few errors like "Inboundaggregator - Error reading last check point"  

 

Those errors are normal. Could you try starting the minemeld service (sudo service minemeld start) and then upload the /opt/minemeld/log/supervisord.log  file ?

Please find the below 

 

2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-web.conf" during parsing
2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2016-07-29 11:14:53,701 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2016-07-29 11:14:53,722 INFO RPC interface 'supervisor' initialized
2016-07-29 11:14:53,722 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-07-29 11:14:53,723 INFO daemonizing the supervisord process
2016-07-29 11:14:53,724 INFO supervisord started with pid 21398
2016-07-29 11:14:54,727 INFO spawned: 'minemeld-engine' with pid 21421
2016-07-29 11:14:54,729 INFO spawned: 'minemeld-traced' with pid 21422
2016-07-29 11:14:54,731 INFO spawned: 'minemeld-web' with pid 21423
2016-07-29 11:14:56,724 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 11:14:56,724 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 11:14:56,724 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:05:35,881 INFO stopped: minemeld-traced (exit status 0)
2016-07-29 12:05:37,826 INFO stopped: minemeld-web (exit status 0)
2016-07-29 12:15:35,931 WARN killing 'minemeld-engine' (21421) with SIGKILL
2016-07-29 12:15:35,951 INFO stopped: minemeld-engine (terminated by SIGKILL)
2016-07-29 12:15:36,039 WARN received SIGTERM indicating exit request
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-web.conf" during parsing
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-traced.conf" during parsing
2016-07-29 12:16:06,230 WARN Included extra file "/opt/minemeld/local/supervisor/config/conf.d/minemeld-engine.conf" during parsing
2016-07-29 12:16:06,274 INFO RPC interface 'supervisor' initialized
2016-07-29 12:16:06,274 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-07-29 12:16:06,275 INFO daemonizing the supervisord process
2016-07-29 12:16:06,276 INFO supervisord started with pid 1208
2016-07-29 12:16:07,279 INFO spawned: 'minemeld-engine' with pid 1322
2016-07-29 12:16:07,282 INFO spawned: 'minemeld-traced' with pid 1323
2016-07-29 12:16:07,285 INFO spawned: 'minemeld-web' with pid 1324
2016-07-29 12:16:08,002 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 12:16:08,660 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:16:08,660 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:16:09,055 INFO spawned: 'minemeld-traced' with pid 1520
2016-07-29 12:16:09,381 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 12:16:11,385 INFO spawned: 'minemeld-traced' with pid 1817
2016-07-29 12:16:12,698 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 12:23:06,548 INFO exited: minemeld-engine (exit status 0; expected)
2016-07-29 12:23:07,552 INFO spawned: 'minemeld-engine' with pid 2096
2016-07-29 12:23:08,583 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-07-29 14:27:31,667 INFO exited: minemeld-traced (exit status 1; not expected)
2016-07-29 14:27:32,671 INFO spawned: 'minemeld-traced' with pid 5451
2016-07-29 14:27:33,962 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

It seems that the trace daemon keeps going down. Could you check /opt/minemeld/log/minemeld-traced.log for errors ?

 

Thanks.

Hi,

  sorry for the late response.. here is the log ::

 

2016-07-29T11:14:54 (21422)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T11:14:54 (21422)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T11:14:54 (21422)main.main INFO: mm-traced config: {}
2016-07-29T11:36:13 (21422)storage.__init__ WARNING: MAX_ID key not found in 00000000579a9c80
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/hub.py", line 140, in handle
self.handler(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 75, in _sigterm_handler
raise KeyboardInterrupt('Ctrl-C from _sigterm_handler')
KeyboardInterrupt: Ctrl-C from _sigterm_handler
2016-07-29T12:05:35 (21422)writer.stop INFO: Writer - stop called
2016-07-29T12:05:35 (21422)queryprocessor.stop INFO: QueryProcessor - stop called
2016-07-29T12:05:35 (21422)storage.stop INFO: Store - stop called
2016-07-29T12:16:07 (1323)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:07 (1323)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:07 (1323)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 135, in main
comm.start()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 541, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
2016-07-29T12:16:09 (1520)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:09 (1520)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:09 (1520)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 135, in main
comm.start()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 541, in start
amqp.connection.Connection(**self.amqp_config)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
2016-07-29T12:16:11 (1817)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T12:16:11 (1817)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T12:16:11 (1817)main.main INFO: mm-traced config: {}
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 516, in _ioloop
self._connections[j].drain_events()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 302, in drain_events
chanmap, None, timeout=timeout,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 365, in _wait_multiple
channel, method_sig, args, content = read_timeout(timeout)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/connection.py", line 329, in read_timeout
return self.method_reader.read_method()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
raise m
error: [Errno 110] Connection timed out
<Greenlet at 0x7f279c01a230: <bound method AMQP._ioloop of <minemeld.comm.amqp.AMQP object at 0x7f279c002410>>(0)> failed with error

2016-07-29T14:27:31 (1817)amqp._ioloop_failure ERROR: _ioloop_failure: exception in ioloop
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 522, in _ioloop_failure
g.get()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
raise self._exception
error: [Errno 110] Connection timed out
2016-07-29T14:27:31 (1817)writer.stop INFO: Writer - stop called
2016-07-29T14:27:31 (1817)queryprocessor.stop INFO: QueryProcessor - stop called
2016-07-29T14:27:31 (1817)storage.stop INFO: Store - stop called
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-traced", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 150, in main
_cleanup()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/traced/main.py", line 81, in _cleanup
comm.stop()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 598, in stop
rpcc.disconnect()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 301, in disconnect
self.channel.close()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/channel.py", line 176, in close
self._send_method((20, 40), args)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/socket.py", line 460, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/gevent/socket.py", line 437, in send
return sock.send(data, flags)
socket.error: [Errno 32] Broken pipe
2016-07-29T14:27:32 (5451)main.main INFO: Starting mm-traced version 0.9.18
2016-07-29T14:27:32 (5451)main.main INFO: mm-traced arguments: Namespace(config='/opt/minemeld/local/config/traced.yml', verbose=False)
2016-07-29T14:27:32 (5451)main.main INFO: mm-traced config: {}

I made a mistake in the Ubuntu installation instructions. Could you try the following commands and then restart the instance  ?

 

sudo ip6tables -A INPUT -i lo -j ACCEPT
sudo ip6tables -P OUTPUT ACCEPT
sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"

 

I fixed the article as well.

 

Thanks !

Hi lmori,

   I tried the above commands and i am still getting the same error as Bad Gateway.. I guess there is something to do with minemeld.sock file in /opt/minemeld/local/supervisor/run    or   minemeld.sock file in /opt/minemeld/local/supervisor/config files.. I am not sure. Looking for your help. 

 

thanks  

 

 

Hi maltwist,

- have you rebooted the instance after applying the changes ?

- could you attach the output of the following commands ?

lsb_release -r -c
sudo netstat -anp
sudo dpkg -l

Thanks,

luigi

Hi lmori.

  Yes i rebooted.. i ran the above commands and it says all the sream status are connected.. May i know how to start/restart the Minemeld instance. ?

 

Also when i try to check 

sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.

 

It syas minemeld.sock refused connection.. 

 

your help will be highly appericiated.. 

regards

Maltwist

Hi MalTwist,

could you attach the output of those commands or send it to me ?

 

Thanks,

luigi 

Hi lmori,

  Please find the attached files.

pic 1 - is the reply i got for the  command : 

 

lsb_release -r -c
sudo netstat -anp
sudo dpkg -l 

 Pic 2 : Is the reply for the command : 

 

$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord. 

 we need to note that status says : minemeld.sock   refused connection. 

 

 

pic 3 :  i suspect some issue on the configuration file and the minemeld.sock  file.. 

 

 

thanks 

  • 18427 Views
  • 15 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!