Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

[mboehlke]

We've been working on getting the syslog miner working to block IPs from the threat logs. However, we want them to stay on the block list for longer than the default 1 hour. From reading through the prototype customization documentation, I think I should be able to configure a prototype somethink like this:

source_name: panos.syslog
age_out:
    default: last_seen+7d
    sudden_death: false
    interval: 1800
attributes:
    confidence: 100

Which works and the prototype is saved. However, when I add a miner from this prototype and commit the changes, the MineMeld engine refuses to start. It pegs the CPU, retries several times, and then goes into an error state. I've tried this several times and received different errors in the log, but this is the most recent:

2017-03-01T12:44:24 (3482)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
    config=config
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 390, in __init__
    super(SyslogMiner, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 198, in __init__
    self.configure()
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 407, in configure
    self.age_out[k] = parse_age_out(v)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/utils.py", line 175, in parse_age_out
    toks = s.split('+', 1)
AttributeError: 'bool' object has no attribute 'split'
Process Process-1:

Am I mis-configuring the prototype? 

Thank you!

[lmori]

Hi @mboehlke,

in your minemeld-engine.log file you should have line looking like:

2017-02-23T17:12:21 (5002)launcher.main INFO: mm-run.py config: [...]

Could you share it ?

Thanks,

luigi

[mboehlke]

Here you go, @lmori

2017-03-01T12:44:59 (3502)launcher.main INFO: mm-run.py config: _Config(nodes={'BinaryDefense_Artillery_Blocklist': {'inputs': [], 'config': {'url': 'https://www.binarydefense.com/banlist.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 50, 'share_level': 'green'}, 'source_name': 'binarydefense.banlist', 'ignore_regex': '^#.*'}, 'class': 'minemeld.ft.http.HttpFT', 'output': True}, 'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 53, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4', 'BinaryDefense_Artillery_Blocklist'], 'indicator_types': ['IPv4'], 'node_type': 'processor', 'output': True, 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT'}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[_ConfigChange(nodename=u'PAN_syslogMiner-HC', nodeclass=u'minemeld.ft.syslog.SyslogMiner', change=1, detail={'inputs': [], 'config': {'attributes': {'confidence': 100}, 'source_name': 'panos.syslog', 'age_out': {'default': 'last_seen+7d', 'sudden_death': False, 'interval': 1800}}, 'class': 'minemeld.ft.syslog.SyslogMiner', 'output': True})])

[lmori · Accepted solution]

Hi @mboehlke,

thanks. You should remove the sudden_death line from the age_out stanza in the prototype as sudden_death is not supported in the syslog miner.

luigi