About DanWoodruff

Hi!   0.9.26 is a great update - thanks for the hard work. Already using the CIF miner and enjoying the live UI updates.   I'm seeing a a bunch of these errors in my engine log:     2016-10-28T21:31:15 (30348)amqp._callback ERROR: Exception in handling update on topic proc-URL-name-HighConfidence with params {u'source': u'proc-URL-name-HighConfidence', u'indicator': u'http://steponnopets.com.sg/acct/identity.php/?amp=&processing_transaction_ids\xc3\xa3\xc2\xa4a0bf64c8ba839a7a7015=&sessargs=g_5rxbrvaxu_bmhpswnni8z2g3zho1ox', u'value': {u'confidence': 85, u'cif_group': [u'general.name'], u'cif_description': u'wells fargo personal banking', u'cif_tlp': u'privileged', u'share_level': u'green', u'cif_lasttime': u'2016-10-22T22:20:30Z', u'sources': [u'bl-URL-name-Phishing'], u'cif_provider': u'apwg.org', u'cif_altid_tlp': u'restricted', u'cif_tags': [u'phishing'], u'first_seen': 1477704643118L, u'type': u'URL', u'cif_reporttime': u'2016-10-22T22:24:07Z', u'last_seen': 1477704643118L}} Traceback (most recent call last): File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 358, in _callback m(**params) File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter f(self, *args, **kwargs) File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 494, in update value=fltvalue File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter f(self, *args, **kwargs) File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 920, in filtered_update self._add_indicator(now, indicator, value) File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 842, in _add_indicator indicator UnicodeEncodeError: 'ascii' codec can't encode characters in position 77-80: ordinal not in range(128)     The Processor seems to be working and just skipping over these invalid entries, so that's good. This is a new CIF added feed.   Just wanted to report it.   Thanks again, Dan ... View more

  Hi,   I have each of my processors set up to send to two different output miners - one for the firewalls to ingest (stdlib.feedHCGreen) and one for my SIEM to ingest (stdlib.taxiiDataFeed). I'm seeing differences between what is ending up in my SIEM and what is on the list for the firewalls. I believe feedHCGreen outputs are correct while TAXII outputs are off.   Looking at the NODES screen, the number of indicators is different between the output miners. This is consistent with 6 different processors (total of 12 miners, 6 of feedHCGreen and 6 of taxiiDataFeed). Screenshots are below showing differences in counts of indicators.    It is quite possible that I am not understanding how TAXII works or I am doing something wrong. Is it part of the protocol that indicators automatically age out after a period of time? I didn't see anything in the prototype configs that specify an age out time. The result I am trying to accomplish is both sets of outputs having the exact same data at all times.   Thanks! Dan     ... View more

Hi,   As of yesterday morning I'm getting an error in my MineMeld engine log that is indicating too many open files. Also, some of my input and output miners are not updating (they show 0 entries even though there are indicators in the input table in the web interface). The timing indicates these errors are possibly the cause. Examples:     2016-10-23T07:43:43 (23610)basepoller._run ERROR: Exception in polling loop for <feed name>: IO error: <feed name>/194800.ldb: Too many open files IOError: IO error: <feed name>/194800.ldb: Too many open files OSError: [Errno 24] Too many open files: '/opt/minemeld/local/config/<feed name>_indicators.yml.lock' I've restarted MineMeld and the error has gone away for now. It was only running for about 3 days before the errors started:   2016-10-20T09:28:18 (23604)launcher.main INFO: Starting mm-run.py version 0.9.24 Any guidance on how to best increase the max number of file descriptors for MineMeld?   Thanks, Dan     ... View more

Hi there, Is there any guidance for how to set up TAXII output for QRadar to ingest? I see in the latest release notes:   - TAXII DataFeed now translated IP Ranges into CIDR for better compatibility with 3rd party TAXII clients (read IBM QRadar)     So I figure it must be possible 🙂 but when I put the discover service URL into the Threat Intelligence app (https://<hostname>/taxii-discovery-service) I get a very generic error of:   "There is a problem connecting to the TAXII server. Please check your connection information and verify that the TAXII server is available"   In MineMeld I've setup an output node of type stdlib.taxiiDataFeed with an input of one of the aggregators. I'm trying to figure out how to get more detailed error logs from QRadar in the mean time...   Thanks in advance! Dan   ... View more