Minemeld miners timing out, but curl works fine.

Reply
L1 Bithead

Minemeld miners timing out, but curl works fine.

Hi,

 

I am having problems running miners in my network. We use a proxy, so that might be an issue, but the HTTP_PROXY and HTTPS_PROXY values are set correctly, and curl/wget work fine. But I am unsure how to check what Minemeld is doing.

 

This is what I find in the error log when trying to manually trigger "IPNode4" to retrieve its indicators:

 

2017-06-19T16:35:04 (26737)basepoller.hup INFO: IPNode4 - hup received, force polling
2017-06-19T16:35:04 (26737)basepoller._huppable_wait INFO: hup is clear: False
2017-06-19T16:35:04 (26737)basepoller._actor_loop INFO: IPNode4 - command: 1497882904343 poll
2017-06-19T16:35:04 (26737)basepoller._polling_loop INFO: Polling IPNode4
2017-06-19T16:35:04 (26737)connectionpool._new_conn INFO: Starting new HTTPS connection (1): lists.blocklist.de
2017-06-19T16:35:09 (26737)basepoller._poll ERROR: Exception in polling loop for IPNode4: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 720, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 570, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 191, in _build_iterator
**rkwargs
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)

 

However, when trying the url specified for "lists.blocklist.de" from the command line, everything works fine...

 

Any ideas?

L7 Applicator

Re: Minemeld miners timing out, but curl works fine.

Hi @ArildSaether,

is this the only Miner configured ? is this the only Miner with this issue ?

 

Thanks,

luigi

L1 Bithead

Re: Minemeld miners timing out, but curl works fine.

Hi,

 

Not the only one. Some O365 ones work (I got that config as a hand-me-down, and they seem slightly different), but when adding a few IP-based ones, I get this error message for this miner, and for another one. And that one also works from curl.

 

This is the error message from the other miner:

 

2017-06-20T07:51:39 (26737)basepoller._polling_loop INFO: Polling AnotherIPsource
2017-06-20T07:51:39 (26737)connectionpool._new_conn INFO: Starting new HTTPS connection (1): feodotracker.abuse.ch
2017-06-20T07:51:39 (26737)basepoller._poll ERROR: Exception in polling loop for AnotherIPsource: __str__ returned non-string (type SysCallError)
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 720, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/core/minemeld/ft/basepoller.py", line 570, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/core/minemeld/ft/http.py", line 191, in _build_iterator
**rkwargs
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/opt/minemeld/engine/current/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
if 'timed out' in str(err) or 'did not complete (read)' in str(err): # Python 2.6
TypeError: __str__ returned non-string (type SysCallError)
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 sudden_death
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 age_out
2017-06-20T07:51:41 (26737)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-06-20T07:51:41 (26737)basepoller._actor_loop INFO: AnotherIPsource - command: 1497937894247 gc
2017-06-20T07:51:41 (26737)table._query_by_index INFO: Deleted in scan of _withdrawn: 0

L7 Applicator

Re: Minemeld miners timing out, but curl works fine.

Hi @ArildSaether,

are the Miners throwing the errors based on standard prototypes or custom prototypes ?

 

Thanks,

luigi

L1 Bithead

Re: Minemeld miners timing out, but curl works fine.

Hi,

 

Standard. I was just setting them up to make sure I understood things properly. Miner, aggregator and output nodes. Looks good in the graphical visualization, but this error message isn't exactly ideal.

If it hadn't been for the fact that curl works, I would have suspected our proxies (struggled a bit with them when installing Minemeld).

L1 Bithead

Re: Minemeld miners timing out, but curl works fine.

I now firmly believe this is related to the proxies. When monitoring traffic using tcpdump, we see that Minemeld never sends it to the system-wide proxies, it just tries to reach the Internet directly. Should it not use the environment settings?

 

The error message seems to be related to the PA firewalls responding, not the expected web sites.

L7 Applicator

Re: Minemeld miners timing out, but curl works fine.

Hi @ArildSaether,

have you already defined the http proxy env variables in /etc/default/minemeld file ?

Try this:

$ sudo su -
# echo "export HTTP_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# echo "export HTTPS_PROXY=http://10.1.1.1:8080" >> /etc/default/minemeld
# service minemeld stop
 * Stopping: minemeld [ OK ]
# service minemeld start
 * Starting: minemeld [ OK ]
# exit
L1 Bithead

Re: Minemeld miners timing out, but curl works fine.

This works great with Ubuntu 14.04. Both when installing using the apt-packages or when using git/ansible. No problem at all.

 

However, there is no way I am able to make it work with CentOS 7 (ansible install). The proxy settings simply do not seem to get picked up.

L7 Applicator

Re: Minemeld miners timing out, but curl works fine.

Hi @ArildSaether,

I have added an answer in the other thread: https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-behind-corporate-proxy/m-p/163217...

Can we close this and use only the other one ? Just to avoid confusion :-)

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!