Minemeld with Proxy

Reply
Highlighted
L3 Networker

Minemeld with Proxy

Is there any way to perform the minemeld install from behind a proxy? I am deploying a minemeld node in a datacenter where internet access is only available via squid proxy.

 

Thanks,

Nasir

Tags (2)
L7 Applicator

Re: Minemeld with Proxy

Hi Nasir,

yes, you can use HTTP_PROXY and HTTPS_PROXY environment variables in /etc/default/minemeld to specify the proxy address:

 

 

$ sudo su -
# echo "export HTTP_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# echo "export HTTPS_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# service minemeld stop
 * Stopping: minemeld [ OK ] # service minemeld start
* Starting: minemeld [ OK ] # exit

 

L3 Networker

Re: Minemeld with Proxy

Thanks Luigi!

 

However, that'll only work after the initial install, correct? When first booting the VM and logging in, I find that it cannot complete the "initalizing minemeld....this could take sever minutes, please wait" step when sitting behind the proxy. If I interrupt that process in order to update the proxy configuration, I find that the minemeld service was never installed. Is there a way to manually install/re-install minemeld if it has been interrupted?

 

NOTE: I also updated /etc/environment with our proxy configuration and found that things like curl and wget work. Does minemeld check /etc/environment as well, or do I need to also set these values under /etc/default/minemeld?

L7 Applicator

Re: Minemeld with Proxy

Hi Nasir,

you need to export the variables in /etc/default/minemeld, otherwise minemeld service will not use those variables.

 

To restart minemeld installation you can try this:

$ sudo su -
# export HTTP_PROXY=<http proxy address here>
# export HTTPS_PROXY=<https proxy address here>
# rm -rf /var/lib/cloud/*
# cloud-init init
# cloud-init modules -m final
# exit

 There are some packages installed via apt, and I am not 100% sure apt-get will hinerit the environment variables.

L3 Networker

Re: Minemeld with Proxy

Or maybe from a different angle... is there a way to preconfigure the proxy settings by modifying/regenerating the ISO that is attached to the VM?

L3 Networker

Re: Minemeld with Proxy

You're right, there were a few packages that had to be installed. In the end, here was the working order of operations:

 

1) Booted Minemeld after deploying OVA from here:

https://minemeld-dist.s3.amazonaws.com/0_9/minemeld-vm-0.9.0rc2.ova

 

2) Logged in with default credentials

ubuntu/rsplizardspock

 

3) Interrupted Minemeld Intiatlization:

Ctrl+C

 

4) Updated proxy configuration (maybe overkill)

# printf "export HTTP_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld
# printf "export HTTPS_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld

# printf "Acquire::http::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::https::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::ftp::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "http_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "https_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "ftp_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "no_proxy=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment
# printf "HTTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "HTTPS_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "FTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "NO_PROXY=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment

 

5) Cleared old cloud config:

# rm -rf /var/lib/cloud/*

 

6) Installed required packages

 # sudo apt-get update && sudo apt-get install collectd-core nginx rabbitmq-server redis-server librabbitmq1 libsnappy1 libleveldb1 -y
 # apt-get -f install

7) Ran cloud-init stuff

 # cloud-init init
 # cloud-init modules -m final

 

8) Ran minemeld auto-update:

 # minemeld-auto-update 

 

9) Reboot

 # reboot now

L3 Networker

Re: Minemeld with Proxy

Luigi,

 

Looks like Office365 polling may not be using the proxy config from /etc/environment, /etc/default/minemeld:

 

ubuntu@minemeld:~$ tail -n 40 /opt/minemeld/log/minemeld-engine.log | grep ConnectTimeout
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887dcd0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:05 (1412)basepoller._run ERROR: Exception in polling loop for Office365-IPv4-Miner: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:26 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8879c90>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:29 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff190>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:47 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cffc50>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:49 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff9d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:07 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff110>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:10 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff790>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml

 

ubuntu@minemeld:~$ cat /etc/default/minemeld (obfuscated)
export HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
export HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"

 

ubuntu@minemeld:~$ cat /etc/environment (obfuscated)
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
https_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
ftp_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
FTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"

 

Any chance there's an additional place we need to add some proxy awareness?

L7 Applicator

Re: Minemeld with Proxy

Hi Nasir,

please, could you double check with a curl or looking at the proxy logs ?

Sometime the logs of the HTTP library used by MineMeld are misleading when it comes to HTTPS.

 

Luigi

L3 Networker

Re: Minemeld with Proxy

Good idea. Curls and Wget's use the proxy and yield good results. See below (truncated for sanity):

 

ubuntu@minemeld:~$ curl https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2

<!DOCTYPE html>
<html lang="en-US" dir="ltr" class="no-js">
<head>

<title>Office 365 URLs and IP address ranges - Office 365</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<meta http-equiv="content-language" name="Language" content="en-US"/>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
<link href="/favicon.ico?b=16069262150" rel="shortcut icon" type="image/x-icon"/>

<script type="text/javascript">
var ariaToken = '1b3c419f862842df99f49c661508cc63-88b16260-a8e4-4d5d-8055-a5014e02c560-7464';
</script>

Note that other miners (like Spamhaus) are working. Does the office-365 miner config require some additional config that is present in the other miners?

 

-Nasir

 

 

L7 Applicator

Re: Minemeld with Proxy

Created minemeld-core bug #11 to track this.

Related to https://github.com/kennethreitz/requests/issues/2807

 

Should be fixed in the next minor.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!