MineMeld engine:fatal message

[johnsonto]

I'm getting the below message in my minemeld logs and not sure what is causing it  

2018-07-11T00:30:28 (16652)config._destroy_old_nodes INFO: Destroyed nodes: [_ConfigChange(nodename=u'Amazon_IPv4_Agg_General', nodeclass=u'minemeld.ft.ipop.AggregateIPv4FT', change=1, detail={'inputs': ['Amazon_AWS', 'Amazon_CloudFront', 'Amazon_EC2', 'Amazon_S3', 'Amazon_Route53_Agg'], 'config': {'whitelist_prefixes': ['Amazon', 'wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'"], 'name': 'accept IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}), _ConfigChange(nodename=u'compromisedIPs-1531162062986', nodeclass=u'minemeld.ft.http.HttpFT', change=1, detail={'inputs': [], 'config': {'url': 'https://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 50, 'share_level': 'green'}, 'source_name': 'ET.compromised_ips'}, 'class': 'minemeld.ft.http.HttpFT', 'output': True}), _ConfigChange(nodename=u'wood_IPagg', nodeclass=u'minemeld.ft.ipop.AggregateIPv4FT', change=1, detail={'inputs': ['compromisedIPs-1531162062986', 'blockIPs-1531162050810'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'"], 'name': 'accept IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}), _ConfigChange(nodename=u'Amazon_Route53_Agg', nodeclass=u'minemeld.ft.ipop.AggregateIPv4FT', change=1, detail={'inputs': ['Amazon_ROUTE53', 'Amazon_Route53_HealthChecks'], 'config': {'whitelist_prefixes': ['Amazon', 'wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'"], 'name': 'accept IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}), _ConfigChange(nodename=u'feedMCGreen-1531163955644', nodeclass=u'minemeld.ft.redis.RedisSet', change=1, detail={'inputs': ['wood_IPagg'], 'indicator_types': ['any'], 'node_type': 'output', 'output': False, 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet'}), _ConfigChange(nodename=u'blockIPs-1531162050810', nodeclass=u'minemeld.ft.http.HttpFT', change=1, detail={'inputs': [], 'config': {'url': 'http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt', 'attributes': {'confidence': 50, 'type': 'IPv4', 'share_level': 'green'}, 'source_name': 'ET.block_ips', 'ignore_regex': '^#'}, 'class': 'minemeld.ft.http.HttpFT', 'output': True}), _ConfigChange(nodename=u'wood_output', nodeclass=u'minemeld.ft.redis.RedisSet', change=1, detail={'inputs': ['wood_IPagg'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}), _ConfigChange(nodename=u'Amazon_Feed', nodeclass=u'minemeld.ft.redis.RedisSet', change=1, detail={'inputs': ['Amazon_IPv4_Agg_General'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False})]
Traceback (most recent call last):
File "/opt/minemeld/engine/current/bin/mm-run", line 11, in <module>
sys.exit(main())
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 218, in main
config = minemeld.run.config.load_config(args.config)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 567, in load_config
return _load_config_from_dir(config_path)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 417, in _load_config_from_dir
_destroy_old_nodes(cconfig)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/run/config.py", line 357, in _destroy_old_nodes
dpool = multiprocessing.Pool()
File "/usr/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 138, in __init__
self._setup_queues()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 234, in _setup_queues
self._inqueue = SimpleQueue()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 352, in __init__
self._rlock = Lock()
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 30] Read-only file system

I have rebooted the system but that hasn't cleared anything up.  

[woodd0]

I work with johnsonto, and can add some context to the error log. 

First, we had two miners aggregating together into an output, and that's been working for some time.

Another two-miner to output pair was made, and while the config had a mismatch between confidence levels (such that nothing was being output), the MineMeld server was still working as expected.

I changed the prototype of the new output node for a lower confidence, and also created a few more independent miners which connected to a new third output. After this change, the above error reared up, and is preventing our MineMeld engine from running. We're able to commit a new config, but when the engine tries to start up, it continues to crash, always on the same "OSError: Read-only file system". 

So far we've been able to change the config, and that leaves me wondering what exactly is read-only. 

[woodd0]

In case anyone else runs into this, we were unable to find a solution and rolled back the VM hosting the server as an alternate solution.