Minemeld with Proxy

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Minemeld with Proxy

L3 Networker

Is there any way to perform the minemeld install from behind a proxy? I am deploying a minemeld node in a datacenter where internet access is only available via squid proxy.

 

Thanks,

Nasir

3 accepted solutions

Accepted Solutions

You're right, there were a few packages that had to be installed. In the end, here was the working order of operations:

 

1) Booted Minemeld after deploying OVA from here:

https://minemeld-dist.s3.amazonaws.com/0_9/minemeld-vm-0.9.0rc2.ova

 

2) Logged in with default credentials

ubuntu/rsplizardspock

 

3) Interrupted Minemeld Intiatlization:

Ctrl+C

 

4) Updated proxy configuration (maybe overkill)

# printf "export HTTP_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld
# printf "export HTTPS_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld

# printf "Acquire::http::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::https::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::ftp::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "http_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "https_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "ftp_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "no_proxy=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment
# printf "HTTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "HTTPS_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "FTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "NO_PROXY=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment

 

5) Cleared old cloud config:

# rm -rf /var/lib/cloud/*

 

6) Installed required packages

 # sudo apt-get update && sudo apt-get install collectd-core nginx rabbitmq-server redis-server librabbitmq1 libsnappy1 libleveldb1 -y
 # apt-get -f install

7) Ran cloud-init stuff

 # cloud-init init
 # cloud-init modules -m final

 

😎 Ran minemeld auto-update:

 # minemeld-auto-update 

 

9) Reboot

 # reboot now

View solution in original post

Created minemeld-core bug #11 to track this.

Related to https://github.com/kennethreitz/requests/issues/2807

 

Should be fixed in the next minor.

View solution in original post

12 REPLIES 12

L7 Applicator

Hi Nasir,

yes, you can use HTTP_PROXY and HTTPS_PROXY environment variables in /etc/default/minemeld to specify the proxy address:

 

 

$ sudo su -
# echo "export HTTP_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# echo "export HTTPS_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# service minemeld stop
 * Stopping: minemeld [ OK ] # service minemeld start
* Starting: minemeld [ OK ] # exit

 

Thanks Luigi!

 

However, that'll only work after the initial install, correct? When first booting the VM and logging in, I find that it cannot complete the "initalizing minemeld....this could take sever minutes, please wait" step when sitting behind the proxy. If I interrupt that process in order to update the proxy configuration, I find that the minemeld service was never installed. Is there a way to manually install/re-install minemeld if it has been interrupted?

 

NOTE: I also updated /etc/environment with our proxy configuration and found that things like curl and wget work. Does minemeld check /etc/environment as well, or do I need to also set these values under /etc/default/minemeld?

Hi Nasir,

you need to export the variables in /etc/default/minemeld, otherwise minemeld service will not use those variables.

 

To restart minemeld installation you can try this:

$ sudo su -
# export HTTP_PROXY=<http proxy address here>
# export HTTPS_PROXY=<https proxy address here>
# rm -rf /var/lib/cloud/*
# cloud-init init
# cloud-init modules -m final
# exit

 There are some packages installed via apt, and I am not 100% sure apt-get will hinerit the environment variables.

Or maybe from a different angle... is there a way to preconfigure the proxy settings by modifying/regenerating the ISO that is attached to the VM?

You're right, there were a few packages that had to be installed. In the end, here was the working order of operations:

 

1) Booted Minemeld after deploying OVA from here:

https://minemeld-dist.s3.amazonaws.com/0_9/minemeld-vm-0.9.0rc2.ova

 

2) Logged in with default credentials

ubuntu/rsplizardspock

 

3) Interrupted Minemeld Intiatlization:

Ctrl+C

 

4) Updated proxy configuration (maybe overkill)

# printf "export HTTP_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld
# printf "export HTTPS_PROXY=http://username:password@10.1.1.1:8080\n" >> /etc/default/minemeld

# printf "Acquire::http::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::https::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "Acquire::ftp::proxy \"http://username:password@10.1.1.1:8080/\";\n" >> /etc/apt/apt.conf.d/95proxies

# printf "http_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "https_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "ftp_proxy=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "no_proxy=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment
# printf "HTTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "HTTPS_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "FTP_PROXY=\"http://username:password@10.1.1.1:8080/\"\n" >> /etc/environment
# printf "NO_PROXY=\"localhost,127.0.0.1,localaddress,.localdomain.com\"\n" >> /etc/environment

 

5) Cleared old cloud config:

# rm -rf /var/lib/cloud/*

 

6) Installed required packages

 # sudo apt-get update && sudo apt-get install collectd-core nginx rabbitmq-server redis-server librabbitmq1 libsnappy1 libleveldb1 -y
 # apt-get -f install

7) Ran cloud-init stuff

 # cloud-init init
 # cloud-init modules -m final

 

😎 Ran minemeld auto-update:

 # minemeld-auto-update 

 

9) Reboot

 # reboot now

Luigi,

 

Looks like Office365 polling may not be using the proxy config from /etc/environment, /etc/default/minemeld:

 

ubuntu@minemeld:~$ tail -n 40 /opt/minemeld/log/minemeld-engine.log | grep ConnectTimeout
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887dcd0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:05 (1412)basepoller._run ERROR: Exception in polling loop for Office365-IPv4-Miner: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host='support.content.office.net', port=443): Max retries exceeded with url: /en-us/static/O365IPAddresses.xml (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a887d6d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)'))
2016-05-03T15:48:26 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8879c90>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:29 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff190>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:47 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cffc50>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:48:49 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff9d0>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:07 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff110>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml
2016-05-03T15:49:10 (1412)connectionpool.urlopen WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe4a8cff790>, 'Connection to support.content.office.net timed out. (connect timeout=20)')': /en-us/static/O365IPAddresses.xml

 

ubuntu@minemeld:~$ cat /etc/default/minemeld (obfuscated)
export HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
export HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"

 

ubuntu@minemeld:~$ cat /etc/environment (obfuscated)
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
https_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
ftp_proxy="http://xxxx:yyyy@proxy.mycustomer:8080/"
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
HTTPS_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
FTP_PROXY="http://xxxx:yyyy@proxy.mycustomer:8080/"
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"

 

Any chance there's an additional place we need to add some proxy awareness?

Hi Nasir,

please, could you double check with a curl or looking at the proxy logs ?

Sometime the logs of the HTTP library used by MineMeld are misleading when it comes to HTTPS.

 

Luigi

Good idea. Curls and Wget's use the proxy and yield good results. See below (truncated for sanity):

 

ubuntu@minemeld:~$ curl https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2

<!DOCTYPE html>
<html lang="en-US" dir="ltr" class="no-js">
<head>

<title>Office 365 URLs and IP address ranges - Office 365</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<meta http-equiv="content-language" name="Language" content="en-US"/>
<meta http-equiv="X-UA-Compatible" content="IE=edge"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
<link href="/favicon.ico?b=16069262150" rel="shortcut icon" type="image/x-icon"/>

<script type="text/javascript">
var ariaToken = '1b3c419f862842df99f49c661508cc63-88b16260-a8e4-4d5d-8055-a5014e02c560-7464';
</script>

Note that other miners (like Spamhaus) are working. Does the office-365 miner config require some additional config that is present in the other miners?

 

-Nasir

 

 

Created minemeld-core bug #11 to track this.

Related to https://github.com/kennethreitz/requests/issues/2807

 

Should be fixed in the next minor.

Fixed in MineMeld 0.9.10 (just released)

I can confirm that 0.9.10 fixes this issue for us. Now O365 prototypes also use our squid proxy.

Just a quick note, that this doesn't work on Ubuntu 16.04, as this version uses systemd, and after some small troubleshooting @lmori came up with the following solution for the problem:

 

@borising ok, now I understand why it's not working: Ubuntu 16.04 LTS uses systemd. To add the environment variables to MineMeld you should do `sudo systemctl edit minemeld.service` and then paste something like


luigimori [10:29 PM]
added this Plain Text snippet: Untitled 
[Service]
Environment=HTTPS_PROXY=<...>
Environment=HTTP_PROXY=<...>

luigimori [10:29 PM]
`sudo systemctl daemon-reload`
`sudo systemctl restart minemeld`

And after I did this, it worked as expected.

  • 3 accepted solutions
  • 16425 Views
  • 12 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!