topic Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway" in General Topics

Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 15:45:17 GMT

Our Minemeld will longer let me log in via the user interface. The operating system logon works fine.

When trying to log into the web interface I get "ERROR CHECKING CREDENTIALS - Bad Gateway"

We are a Windows shop and don't really have any Linux skills to breing to bear on this.

What do I need to do to resolve this? It is a production impacting matter.

Thanks,

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 15:54:28 GMT

Hi @DwightH,

probably one (or many) MineMeld services is down. Can you run the following command to check them?

ubuntu@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 15:57:27 GMT

Hi,

The response is

unix:///opt/minemeld/local/supervisor/run/minemeld.sock refused connection

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 16:00:59 GMT

Everything looks down, then.

What about a full MineMeld service restart?

sudo service minemeld restart

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 16:04:11 GMT

ubuntu@minemeld:~$ sudo service minemeld restart
* Restarting: minemeld unix:///opt/minemeld/local/supervisor/run/minemeld.sock refused connection
Unlinking stale socket /opt/minemeld/local/supervisor/run/minemeld.sock
[ OK ]
ubuntu@minemeld:~$

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 16:23:20 GMT

Hi,

So I tried the restart command again and while the app did restart and I can now log in, my office365 output is disabled. This is supposed to feed an External Distribution List in my 3060s to descope all Office365 ips from decryption.

ubuntu@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
unix:///opt/minemeld/local/supervisor/run/minemeld.sock refused connection
ubuntu@minemeld:~$ sudo service minemeld restart
* Restarting: minemeld unix:///opt/minemeld/local/supervisor/run/minemeld.sock refused connection
Unlinking stale socket /opt/minemeld/local/supervisor/run/minemeld.sock
[ OK ]
ubuntu@minemeld:~$ ^C
ubuntu@minemeld:~$ sudo service minemeld restart
* Restarting: minemeld minemeld-traced: stopped
minemeld-web: stopped
minemeld-engine: stopped
[ OK ]
ubuntu@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine RUNNING pid 2174, uptime 0:03:12
minemeld-traced RUNNING pid 2175, uptime 0:03:12
minemeld-web RUNNING pid 2176, uptime 0:03:12
ubuntu@minemeld:~$

office365_IPv4s node shows disabled and there is an empty page on it's url.

Shouldn't this be Enabled? I seem to recall there would be a list of ip blocks on this url to feed into the EDL.

Thanks,

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 16:41:05 GMT

Disabled in the output node is OK. It does not mean that the "output is disable". It means that the "output of the node" is not consumed by any other node (it is a final node).

Zero indicators in an output means that the graph, starting from the miner, is not feeding the pipe. You should navigate to your miner (input node) and click on the recycle icon besides the "last run" message to force a new data poll.

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 16:47:35 GMT

Hi,

Interesting. All of my office365 related miners show Started but the numbers are all 0s. I restarted a few with no seeming change.

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 17:41:30 GMT

SO it looks like 3 days ago the office365_IPv4s NODE emptied out to 0 from a typical 661 addresses.

How does one troubleshoot this and get it working again?

30 day graphic attached.

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 18:06:09 GMT

Sharing the configuration (/opt/minemeld/local/config/running-config.yml) and the engine log (/opt/minemeld/log/minemeld-engine.log) may give us a try to troubleshoot it.

Is there any chance that a network/security policy change is blocking MineMeld from reaching Internet resources?

Try a basic test from the Ubuntu server to verify connectivity:

prompt# curl http://clients1.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 18:06:12 GMT

Sharing the configuration (/opt/minemeld/local/config/running-config.yml) and the engine log (/opt/minemeld/log/minemeld-engine.log) may give us a try to troubleshoot it.

Is there any chance that a network/security policy change is blocking MineMeld from reaching Internet resources?

Try a basic test from the Ubuntu server to verify connectivity:

prompt# curl http://clients1.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 18:21:29 GMT

I get

ubuntu@minemeld:~$ curl http://clients1.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
ubuntu@minemeld:~$

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 18:23:18 GMT

Looks good.

Share, please, configuration and engine log files so we can try understand what's wrong.

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 18:28:30 GMT

Thanks.

Here are the requested files (attached).

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

xhoms — Tue, 23 Jan 2018 18:37:46 GMT

Your engine log file is full of SSL certification validation errors. It is failing to successfully establish communication with your Internet feed providers (Microsoft between them).

Is there any device performing SSL decryption between MineMeld and Internet? Can you turn it off. In such a case you need to import its root CA into MineMeld (https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-SSL-Certificates/m-p/113561#M345)

Re: Minemeld down - Can't log into UI "ERROR CHECKING CREDENTIALS - Bad Gateway"

DwightH — Tue, 23 Jan 2018 18:54:18 GMT

Thanks.

I just checked my Decryption Pre Rules in Palo Alto and noticed Minemeld wasn't there. I added it and pushed to the 3060s.

I restarted Minemeld and at a glance it apprears to be behaving better.

I will keep a close eye on it for the next few hours.

The url http://norminemeld/feeds/office365_IPv4s is populated once again.

Thank you!