'ascii' codec can't encode characters

[DanWoodruff]

Hi!

0.9.26 is a great update - thanks for the hard work. Already using the CIF miner and enjoying the live UI updates.

I'm seeing a a bunch of these errors in my engine log:

2016-10-28T21:31:15 (30348)amqp._callback ERROR: Exception in handling update on topic proc-URL-name-HighConfidence with params {u'source': u'proc-URL-name-HighConfidence', u'indicator': u'http://steponnopets.com.sg/acct/identity.php/?amp=&processing_transaction_ids\xc3\xa3\xc2\xa4a0bf64c8ba839a7a7015=&sessargs=g_5rxbrvaxu_bmhpswnni8z2g3zho1ox', u'value': {u'confidence': 85, u'cif_group': [u'general.name'], u'cif_description': u'wells fargo personal banking', u'cif_tlp': u'privileged', u'share_level': u'green', u'cif_lasttime': u'2016-10-22T22:20:30Z', u'sources': [u'bl-URL-name-Phishing'], u'cif_provider': u'apwg.org', u'cif_altid_tlp': u'restricted', u'cif_tags': [u'phishing'], u'first_seen': 1477704643118L, u'type': u'URL', u'cif_reporttime': u'2016-10-22T22:24:07Z', u'last_seen': 1477704643118L}}
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 358, in _callback
m(**params)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 494, in update
value=fltvalue
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 920, in filtered_update
self._add_indicator(now, indicator, value)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 842, in _add_indicator
indicator
UnicodeEncodeError: 'ascii' codec can't encode characters in position 77-80: ordinal not in range(128)

The Processor seems to be working and just skipping over these invalid entries, so that's good. This is a new CIF added feed.

Just wanted to report it.

Thanks again,

Dan

[lmori · Accepted solution]

Hi @DanWoodruff,

thanks for reporting this, I think the problem is related to the non-ascii character in the URL ('\xc3...'). I have opened bug minemeld-core#77 (https://github.com/PaloAltoNetworks/minemeld-core/issues/77) to track this issue.

I am planning to have it fixed by the next release.

Thanks !

luigi