Minemeld with Proxy

[nbilal]

Is there any way to perform the minemeld install from behind a proxy? I am deploying a minemeld node in a datacenter where internet access is only available via squid proxy.

Thanks,

Nasir

[lmori]

Hi Nasir,

yes, you can use HTTP_PROXY and HTTPS_PROXY environment variables in /etc/default/minemeld to specify the proxy address:

$ sudo su -
# echo "export HTTP_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# echo "export HTTPS_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# service minemeld stop
 * Stopping: minemeld [ OK ]
# service minemeld start
 * Starting: minemeld [ OK ]
# exit

[nbilal]

Thanks Luigi!

However, that'll only work after the initial install, correct? When first booting the VM and logging in, I find that it cannot complete the "initalizing minemeld....this could take sever minutes, please wait" step when sitting behind the proxy. If I interrupt that process in order to update the proxy configuration, I find that the minemeld service was never installed. Is there a way to manually install/re-install minemeld if it has been interrupted?

NOTE: I also updated /etc/environment with our proxy configuration and found that things like curl and wget work. Does minemeld check /etc/environment as well, or do I need to also set these values under /etc/default/minemeld?

[lmori]

Hi Nasir,

you need to export the variables in /etc/default/minemeld, otherwise minemeld service will not use those variables.

To restart minemeld installation you can try this:

$ sudo su -
# export HTTP_PROXY=<http proxy address here>
# export HTTPS_PROXY=<https proxy address here>
# rm -rf /var/lib/cloud/*
# cloud-init init
# cloud-init modules -m final
# exit

 There are some packages installed via apt, and I am not 100% sure apt-get will hinerit the environment variables.

[nbilal]

Or maybe from a different angle... is there a way to preconfigure the proxy settings by modifying/regenerating the ISO that is attached to the VM?

[nbilal · Accepted solution]

You're right, there were a few packages that had to be installed. In the end, here was the working order of operations:

1) Booted Minemeld after deploying OVA from here:

https://minemeld-dist.s3.amazonaws.com/0_9/minemeld-vm-0.9.0rc2.ova

2) Logged in with default credentials

ubuntu/rsplizardspock

3) Interrupted Minemeld Intiatlization:

Ctrl+C

4) Updated proxy configuration (maybe overkill)

# printf "export HTTP_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld
# printf "export HTTPS_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld

# printf "Acquire::http::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::https::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::ftp::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "http_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "https_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "ftp_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "no_proxy=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment
# printf "HTTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "HTTPS_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "FTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "NO_PROXY=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment

5) Cleared old cloud config:

# rm -rf /var/lib/cloud/*

6) Installed required packages

 # sudo apt-get update && sudo apt-get install collectd-core nginx rabbitmq-server redis-server librabbitmq1 libsnappy1 libleveldb1 -y
 # apt-get -f install

7) Ran cloud-init stuff

 # cloud-init init
 # cloud-init modules -m final

😎 Ran minemeld auto-update:

 # minemeld-auto-update 

9) Reboot

 # reboot now

[nbilal]

Luigi,

Looks like Office365 polling may not be using the proxy config from /etc/environment, /etc/default/minemeld:

ubuntu@minemeld:~$ tail -n 40 /opt/minemeld/log/minemeld-engine.log | grep ConnectTimeout
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887dcd0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:05 (1412)basepoller._run ERROR: Exception in polling loop for Office365-IPv4-Miner: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:26 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8879c90>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:29 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff190>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:47 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cffc50>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:49 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff9d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:07 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff110>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:10 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff790>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml

ubuntu@minemeld:~$ cat /etc/default/minemeld (obfuscated)
export HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
export HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"

ubuntu@minemeld:~$ cat /etc/environment (obfuscated)
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
https_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
ftp_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
FTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"

Any chance there's an additional place we need to add some proxy awareness?

[lmori]

Hi Nasir,

please, could you double check with a curl or looking at the proxy logs ?

Sometime the logs of the HTTP library used by MineMeld are misleading when it comes to HTTPS.

Luigi

[nbilal]

Good idea. Curls and Wget's use the proxy and yield good results. See below (truncated for sanity):

ubuntu@minemeld:~$ curl https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2

<!DOCTYPE html>
<html lang="en-US" dir="ltr" class="no-js">
<head>

<title>Office 365 URLs and IP address ranges - Office 365</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<meta http-equiv="content-language" name="Language" content="en-US"/>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
<link href="/favicon.ico?b=16069262150" rel="shortcut icon" type="image/x-icon"/>

<script type="text/javascript">
var ariaToken = '1b3c419f862842df99f49c661508cc63-88b16260-a8e4-4d5d-8055-a5014e02c560-7464';
</script>

Note that other miners (like Spamhaus) are working. Does the office-365 miner config require some additional config that is present in the other miners?

-Nasir

[lmori · Accepted solution]

Created minemeld-core bug #11 to track this.

Related to https://github.com/kennethreitz/requests/issues/2807

Should be fixed in the next minor.

[lmori · Accepted solution]

Fixed in MineMeld 0.9.10 (just released)

[nbilal]

I can confirm that 0.9.10 fixes this issue for us. Now O365 prototypes also use our squid proxy.

[borising]

Just a quick note, that this doesn't work on Ubuntu 16.04, as this version uses systemd, and after some small troubleshooting @lmori came up with the following solution for the problem:

@borising ok, now I understand why it's not working: Ubuntu 16.04 LTS uses systemd. To add the environment variables to MineMeld you should do `sudo systemctl edit minemeld.service` and then paste something like


luigimori [10:29 PM]
added this Plain Text snippet: Untitled 
[Service]
Environment=HTTPS_PROXY=<...>
Environment=HTTP_PROXY=<...>

luigimori [10:29 PM]
`sudo systemctl daemon-reload`
`sudo systemctl restart minemeld`

And after I did this, it worked as expected.