topic Re: Nodes polling error in General Topics

Nodes polling error

HolgerKiene — Wed, 19 Apr 2017 10:49:03 GMT

Hello somewho have an idea?

Installed Minemeld on an fresh Ubuntu 14.0.4 like the manual installation guide.

Import the Office365 configuration

All Nodes got an SSL Error message see below

2017-04-19T12:45:54 (22890)basepoller.hup INFO: office365_O365 - hup received, force polling
2017-04-19T12:45:54 (22890)basepoller._huppable_wait INFO: hup is clear: False
2017-04-19T12:45:54 (22890)basepoller._actor_loop INFO: office365_O365 - command: 1492598754316 poll
2017-04-19T12:45:54 (22890)basepoller._polling_loop INFO: Polling office365_O365
2017-04-19T12:45:54 (22890)connectionpool._new_conn INFO: Starting new HTTPS connection (1): support.content.office.net
2017-04-19T12:45:54 (22890)basepoller._poll ERROR: Exception in polling loop for office365_O365: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 701, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 568, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 165, in _build_iterator
oiterator = self._o365_iterator(now)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 115, in _o365_iterator
r = _session.send(prepreq, **rkwargs)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

Any guidance that can be provided would be greatly appreciated!

Thanks Holger

Re: Nodes polling error

lmori — Wed, 19 Apr 2017 11:08:09 GMT

Hi @HolgerKiene,

certificate verification is failing. Are you behind a proxy or a device doing SSL decryption ?

Could you open a shell on the MineMeld instance, issue the following and report back any error you see ?

$ cd /tmp/ && wget https://support.content.office.net/en-us/static/O365IPAddresses.xml

Thanks,

luigi

Re: Nodes polling error

HolgerKiene — Wed, 19 Apr 2017 12:24:03 GMT

Thanks for for your fast answer,

You're right my mistake

Holger

Re: Nodes polling error

lmori — Mon, 24 Apr 2017 06:40:56 GMT

Hi @HolgerKiene,

thanks for taking the time to tell us everything is working !

luigi

Re: Nodes polling error

clockhart — Thu, 19 Oct 2017 19:02:43 GMT

Another node get error:

dcadmin@MICMM01:/tmp$ wget https://check.torproject.org/exit-addresses

--2017-10-19 12:54:22-- https://check.torproject.org/exit-addresses

Resolving check.torproject.org (check.torproject.org)... 146.112.61.106, ::ffff:146.112.61.106

Connecting to check.torproject.org (check.torproject.org)|146.112.61.106|:443... connected.

ERROR: cannot verify check.torproject.org's certificate, issued by ‘/CN=Cisco Umbrella Secondary SubCA dfw-SG/O=Cisco’:

Unable to locally verify the issuer's authority.

To connect to check.torproject.org insecurely, use `--no-check-certificate'.

dcadmin@MICMM01:/tmp$

Can I change the prototype to request http rather than https?

Tor Exit Node:

https://check.torproject.org/exit-addresses

Re: Nodes polling error

xhoms — Thu, 19 Oct 2017 19:36:47 GMT

@clockhart : are you aware of the hailataxii.guest_blutmagie_de_torExits prototype in the standard library that also "mines" the tor exit nodes? Any reason not to use it?

I've just realized you're receiving a certificate error from Cisco Umbrella. That means that your MineMeld instance is using a secure proxy to reach the feed (SSL man-in-the-middle). In such a case you need to import the related certificates in the MineMeld's trust ring.

Re: Nodes polling error

clockhart — Thu, 19 Oct 2017 19:39:19 GMT

Good point but my Office365 https requests work behind same DNS proxy. I believe customer is using OpenDNS so that makes sense. I'll take a look at the other prototype to see if I get the same error. I appreciate the response.

Re: Nodes polling error

clockhart — Thu, 19 Oct 2017 20:47:10 GMT

Set up my miner, aggregator and output nodes but no luck. hailataxi Miner reports 273 indicators, which is considerably lower than the tor-exit.nodes (913). Is there a reason for the discrepancy?

Also am I using the wrong aggregator? My list is empty.

Re: Nodes polling error

lmori — Mon, 23 Oct 2017 07:07:17 GMT

Hi @clockhart,

to track tor nodes please use blutmagie.* prototypes, I have found them more reliable over time.

One reason you could considerably less nodes from hailataxii is caused by how TAXII DataFeed work. TAXII DataFeeds are designed to publish updates, not full current lists of indicators. This means that the 273 nodes you see are most probably the 273 tor nodes most recently added to the list of active tor nodes, not the full list. Blutmagie.* and tor.* prototypes instead provide the full current list of Tor nodes.

Hope this helps.

luigi

Re: Nodes polling error

clockhart — Fri, 03 Nov 2017 15:50:00 GMT

Luigi,

Thanks for the response. I'm using blutmagie now for my miner and it's looking good. Appreciate the assistance.

cpl

Re: Nodes polling error

jniedenthal — Tue, 05 Dec 2017 16:30:20 GMT

Hello, I'm just getting started with MineMeld. We have an internal block IP and URL feeds that are hosted on a web server, a text file hosted via HTTPS page. My issue is that this server does not have a valid certificate, but It's my internal server, so I don't care. Is there a way to ignore certificate errors when pulling HTTPS feeds?

Re: Nodes polling error

xhoms — Tue, 05 Dec 2017 16:35:32 GMT

@jniedenthal : That behavior of the HttpFT class is controlled by the verify_cert boolean configuration attibute (defaults to true). Add the attribute to your prototype with the value set to "False"