Nodes polling error

[HolgerKiene]

Hello somewho have an idea?

Installed Minemeld on an fresh Ubuntu 14.0.4 like the manual installation guide.

Import the Office365 configuration 

All Nodes got an SSL Error message see below

2017-04-19T12:45:54 (22890)basepoller.hup INFO: office365_O365 - hup received, force polling
2017-04-19T12:45:54 (22890)basepoller._huppable_wait INFO: hup is clear: False
2017-04-19T12:45:54 (22890)basepoller._actor_loop INFO: office365_O365 - command: 1492598754316 poll
2017-04-19T12:45:54 (22890)basepoller._polling_loop INFO: Polling office365_O365
2017-04-19T12:45:54 (22890)connectionpool._new_conn INFO: Starting new HTTPS connection (1): support.content.office.net
2017-04-19T12:45:54 (22890)basepoller._poll ERROR: Exception in polling loop for office365_O365: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 701, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 568, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 165, in _build_iterator
oiterator = self._o365_iterator(now)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 115, in _o365_iterator
r = _session.send(prepreq, **rkwargs)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/opt/minemeld/engine/0.9.36.post2/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

Any guidance that can be provided would be greatly appreciated! 

Thanks Holger

[lmori]

Hi @HolgerKiene,

certificate verification is failing. Are you behind a proxy or a device doing SSL decryption ?

Could you open a shell on the MineMeld instance, issue the following and report back any error you see ?

$ cd /tmp/ && wget https://support.content.office.net/en-us/static/O365IPAddresses.xml

Thanks,

luigi

[HolgerKiene]

Thanks for for your fast answer,

You're right my mistake

Holger

[lmori]

Hi @HolgerKiene,

thanks for taking the time to tell us everything is working !

luigi

[clockhart]

Another node get error:

dcadmin@MICMM01:/tmp$ wget https://check.torproject.org/exit-addresses

--2017-10-19 12:54:22--  https://check.torproject.org/exit-addresses

Resolving check.torproject.org (check.torproject.org)... 146.112.61.106, ::ffff:146.112.61.106

Connecting to check.torproject.org (check.torproject.org)|146.112.61.106|:443... connected.

ERROR: cannot verify check.torproject.org's certificate, issued by ‘/CN=Cisco Umbrella Secondary SubCA dfw-SG/O=Cisco’:

  Unable to locally verify the issuer's authority.

To connect to check.torproject.org insecurely, use `--no-check-certificate'.

dcadmin@MICMM01:/tmp$

Can I change the prototype to request http rather than https?

Tor Exit Node:

https://check.torproject.org/exit-addresses

[xhoms]

@clockhart : are you aware of the hailataxii.guest_blutmagie_de_torExits prototype in the standard library that also "mines" the tor exit nodes? Any reason not to use it?

I've just realized you're receiving a certificate error from Cisco Umbrella. That means that your MineMeld instance is using a secure proxy to reach the feed (SSL man-in-the-middle). In such a case you need to import the related certificates in the MineMeld's trust ring.

[clockhart]

Good point but my Office365 https requests work behind same DNS proxy. I believe customer is using OpenDNS so that makes sense. I'll take a look at the other prototype to see if I get the same error. I appreciate the response.

[clockhart]

Set up my miner, aggregator and output nodes but no luck. hailataxi Miner reports 273 indicators, which is considerably lower than the tor-exit.nodes (913). Is there a reason for the discrepancy?

Also am I using the wrong aggregator? My list is empty.

[lmori]

Hi @clockhart,

to track tor nodes please use blutmagie.* prototypes, I have found them more reliable over time.

One reason you could considerably less nodes from hailataxii is caused by how TAXII DataFeed work. TAXII DataFeeds are designed to publish updates, not full current lists of indicators. This means that the 273 nodes you see are most probably the 273 tor nodes most recently added to the list of active tor nodes, not the full list. Blutmagie.* and tor.* prototypes instead provide the full current list of Tor nodes.

Hope this helps.

luigi

[clockhart]

Luigi,

Thanks for the response. I'm using blutmagie now for my miner and it's looking good. Appreciate the assistance.

cpl

[jniedenthal]

Hello, I'm just getting started with MineMeld. We have an internal block IP and URL feeds that are hosted on a web server, a text file hosted via HTTPS page. My issue is that this server does not have a valid certificate, but It's my internal server, so I don't care. Is there a way to ignore certificate errors when pulling HTTPS feeds? 

[xhoms]

@jniedenthal : That behavior of the HttpFT class is controlled by the verify_cert boolean configuration attibute (defaults to true). Add the attribute to your prototype with the value set to "False"