Minemeld miners timing out, but curl works fine.

[ArildSaether]

Hi,

I am having problems running miners in my network. We use a proxy, so that might be an issue, but the HTTP_PROXY and HTTPS_PROXY values are set correctly, and curl/wget work fine. But I am unsure how to check what Minemeld is doing.

This is what I find in the error log when trying to manually trigger "IPNode4" to retrieve its indicators:

2017-06-19T16:35:04 (26737)basepoller.hup INFO: IPNode4 - hup received, force polling
2017-06-19T16:35:04 (26737)basepoller._huppable_wait INFO: hup is clear: False
2017-06-19T16:35:04 (26737)basepoller._actor_loop INFO: IPNode4 - command: 1497882904343 poll
2017-06-19T16:35:04 (26737)basepoller._polling_loop INFO: Polling IPNode4
2017-06-19T16:35:04 (26737)connectionpool._new_conn INFO: Starting new HTTPS connection (1): lists.blocklist.de
2017-06-19T16:35:09 (26737)basepoller._poll ERROR: Exception in polling loop for IPNode4: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 720, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 570, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 191, in _build_iterator
**rkwargs
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)

However, when trying the url specified for "lists.blocklist.de" from the command line, everything works fine...

Any ideas?

[lmori]

Hi @ArildSaether,

is this the only Miner configured ? is this the only Miner with this issue ?

Thanks,

luigi

[ArildSaether]

Hi,

Not the only one. Some O365 ones work (I got that config as a hand-me-down, and they seem slightly different), but when adding a few IP-based ones, I get this error message for this miner, and for another one. And that one also works from curl.

This is the error message from the other miner:

2017-06-20T07:51:39 (26737)basepoller._polling_loop INFO: Polling AnotherIPsource
2017-06-20T07:51:39 (26737)connectionpool._new_conn INFO: Starting new HTTPS connection (1): feodotracker.abuse.ch
2017-06-20T07:51:39 (26737)basepoller._poll ERROR: Exception in polling loop for AnotherIPsource: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 720, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 570, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 191, in _build_iterator
**rkwargs
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 sudden_death
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 age_out
2017-06-20T07:51:41 (26737)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 gc
2017-06-20T07:51:41 (26737)table._query_by_index INFO: Deleted in scan of _withdrawn: 0

[lmori]

Hi @ArildSaether,

are the Miners throwing the errors based on standard prototypes or custom prototypes ?

Thanks,

luigi

[ArildSaether]

Hi,

Standard. I was just setting them up to make sure I understood things properly. Miner, aggregator and output nodes. Looks good in the graphical visualization, but this error message isn't exactly ideal.

If it hadn't been for the fact that curl works, I would have suspected our proxies (struggled a bit with them when installing Minemeld).

[ArildSaether]

I now firmly believe this is related to the proxies. When monitoring traffic using tcpdump, we see that Minemeld never sends it to the system-wide proxies, it just tries to reach the Internet directly. Should it not use the environment settings?

The error message seems to be related to the PA firewalls responding, not the expected web sites.

[lmori]

Hi @ArildSaether,

have you already defined the http proxy env variables in /etc/default/minemeld file ?

Try this:

$ sudo su -
# echo "export HTTP_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# echo "export HTTPS_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# service minemeld stop
 * Stopping: minemeld [ OK ]
# service minemeld start
 * Starting: minemeld [ OK ]
# exit

[ArildSaether]

This works great with Ubuntu 14.04. Both when installing using the apt-packages or when using git/ansible. No problem at all.

However, there is no way I am able to make it work with CentOS 7 (ansible install). The proxy settings simply do not seem to get picked up.

[lmori · Accepted solution]

Hi @ArildSaether,

I have added an answer in the other thread: https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-behind-corporate-proxy/m-p/163217...

Can we close this and use only the other one ? Just to avoid confusion 🙂