topic Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting in General Topics

Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

mboehlke — Wed, 01 Mar 2017 13:58:56 GMT

We've been working on getting the syslog miner working to block IPs from the threat logs. However, we want them to stay on the block list for longer than the default 1 hour. From reading through the prototype customization documentation, I think I should be able to configure a prototype somethink like this:

source_name: panos.syslog
age_out:
    default: last_seen+7d
    sudden_death: false
    interval: 1800
attributes:
    confidence: 100

Which works and the prototype is saved. However, when I add a miner from this prototype and commit the changes, the MineMeld engine refuses to start. It pegs the CPU, retries several times, and then goes into an error state. I've tried this several times and received different errors in the log, but this is the most recent:

2017-03-01T12:44:24 (3482)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
    config=config
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 390, in __init__
    super(SyslogMiner, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 198, in __init__
    self.configure()
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 407, in configure
    self.age_out[k] = parse_age_out(v)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/utils.py", line 175, in parse_age_out
    toks = s.split('+', 1)
AttributeError: 'bool' object has no attribute 'split'
Process Process-1:

Am I mis-configuring the prototype?

Thank you!

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

lmori — Thu, 02 Mar 2017 15:44:06 GMT

Hi @mboehlke,

in your minemeld-engine.log file you should have line looking like:

2017-02-23T17:12:21 (5002)launcher.main INFO: mm-run.py config: [...]

Could you share it ?

Thanks,

luigi

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

mboehlke — Mon, 06 Mar 2017 12:14:06 GMT

Here you go, @lmori

2017-03-01T12:44:59 (3502)launcher.main INFO: mm-run.py config: _Config(nodes={'BinaryDefense_Artillery_Blocklist': {'inputs': [], 'config': {'url': 'https://www.binarydefense.com/banlist.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 50, 'share_level': 'green'}, 'source_name': 'binarydefense.banlist', 'ignore_regex': '^#.*'}, 'class': 'minemeld.ft.http.HttpFT', 'output': True}, 'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 53, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4', 'BinaryDefense_Artillery_Blocklist'], 'indicator_types': ['IPv4'], 'node_type': 'processor', 'output': True, 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT'}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[_ConfigChange(nodename=u'PAN_syslogMiner-HC', nodeclass=u'minemeld.ft.syslog.SyslogMiner', change=1, detail={'inputs': [], 'config': {'attributes': {'confidence': 100}, 'source_name': 'panos.syslog', 'age_out': {'default': 'last_seen+7d', 'sudden_death': False, 'interval': 1800}}, 'class': 'minemeld.ft.syslog.SyslogMiner', 'output': True})])

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

lmori — Mon, 06 Mar 2017 12:50:37 GMT

Hi @mboehlke,

thanks. You should remove the sudden_death line from the age_out stanza in the prototype as sudden_death is not supported in the syslog miner.

luigi