Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

Reply
L1 Bithead

Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

We've been working on getting the syslog miner working to block IPs from the threat logs. However, we want them to stay on the block list for longer than the default 1 hour. From reading through the prototype customization documentation, I think I should be able to configure a prototype somethink like this:

source_name: panos.syslog
age_out:
    default: last_seen+7d
    sudden_death: false
    interval: 1800
attributes:
    confidence: 100

Which works and the prototype is saved. However, when I add a miner from this prototype and commit the changes, the MineMeld engine refuses to start. It pegs the CPU, retries several times, and then goes into an error state. I've tried this several times and received different errors in the log, but this is the most recent:

2017-03-01T12:44:24 (3482)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 53, in _run_chassis
    c.configure(fts)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/chassis.py", line 102, in configure
    config=ftconfig.get('config', {})
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 10, in factory
    config=config
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 390, in __init__
    super(SyslogMiner, self).__init__(name, chassis, config)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 198, in __init__
    self.configure()
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 407, in configure
    self.age_out[k] = parse_age_out(v)
  File "/opt/minemeld/engine/0.9.32/local/lib/python2.7/site-packages/minemeld/ft/utils.py", line 175, in parse_age_out
    toks = s.split('+', 1)
AttributeError: 'bool' object has no attribute 'split'
Process Process-1:

Am I mis-configuring the prototype? 

 

Thank you!

L7 Applicator

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

Hi @mboehlke,

in your minemeld-engine.log file you should have line looking like:

2017-02-23T17:12:21 (5002)launcher.main INFO: mm-run.py config: [...]

 

Could you share it ?

 

Thanks,

luigi

L1 Bithead

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

Here you go, @lmori

 

2017-03-01T12:44:59 (3502)launcher.main INFO: mm-run.py config: _Config(nodes={'BinaryDefense_Artillery_Blocklist': {'inputs': [], 'config': {'url': 'https://www.binarydefense.com/banlist.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 50, 'share_level': 'green'}, 'source_name': 'binarydefense.banlist', 'ignore_regex': '^#.*'}, 'class': 'minemeld.ft.http.HttpFT', 'output': True}, 'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 53, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4', 'BinaryDefense_Artillery_Blocklist'], 'indicator_types': ['IPv4'], 'node_type': 'processor', 'output': True, 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT'}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[_ConfigChange(nodename=u'PAN_syslogMiner-HC', nodeclass=u'minemeld.ft.syslog.SyslogMiner', change=1, detail={'inputs': [], 'config': {'attributes': {'confidence': 100}, 'source_name': 'panos.syslog', 'age_out': {'default': 'last_seen+7d', 'sudden_death': False, 'interval': 1800}}, 'class': 'minemeld.ft.syslog.SyslogMiner', 'output': True})])
L7 Applicator

Re: Syslog Miner Prototype Age-out Policy Prevents Engine from Starting

Hi @mboehlke,

thanks. You should remove the sudden_death line from the age_out stanza in the prototype as sudden_death is not supported in the syslog miner.

 

luigi

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!