Error reading last checkpoint

[Forseti]

Hi everyone,

I am facing an issue that floods my output SIEM a little to often. The issue seems to be that the miner node is unable to register where it left of during the last check. Any tips on solving this?

2016-11-14T08:54:30 (17269)base.read_checkpoint ERROR: sslabusech_dyreblacklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 245, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'sslabusech_dyreblacklist.chkp'
2016-11-14T08:54:30 (17269)base.state INFO: sslabusech_dyreblacklist - transitioning to state 1
2016-11-14T08:54:30 (17270)base.read_checkpoint ERROR: sslabusech_ipblacklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 245, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'sslabusech_ipblacklist.chkp'
2016-11-14T08:54:30 (17270)base.state INFO: sslabusech_ipblacklist - transitioning to state 1
2016-11-14T08:54:30 (17270)base.read_checkpoint ERROR: openbl_base - Error reading last checkpoint

Regards,

Forseti

[lmori]

Hi @Forseti,

that ERROR happens only when the engine starts if the checkpoint does not exist. It could happen for 2 reasons:

- the node is new and there is no previous checkpoint to load

- the engine didn't have a "clean" shut down

- could you check before those ERROR messages if you see additional ERRORs ?

- check also the OS syslog and dmesg to see if there was a problem with memory or disk exhaustion

Thanks,

luigi

[Forseti]

Hi Imori, The machine gets rebooted at night because as it seems that the minemeld process isn't stable. (I wasn't able to logon via the web gui after a couple of hours) I can't seem to stop it properly. Neither using 'sudo service mineld stop' nor via supervisord. I've already removed and reinstalled MineMeld completely but the issue seems to persist. However, that seems to be the root of this issue. And ofcourse, now that I try to mimick the behavior, it all goes well.. Will check back when a get a sample! Regards, Jan

[lmori]

I'd like to fix the instability issue first, I think it's related to memory exhaustion.

Would you mind unicast me the logs from /opt/minemeld/log ? My email is lmori@paloaltonetworks.com.

Also please note that when MineMeld start without a checkpoint it starts crunching all the indicators, and it could take a while. Are your running 0.9.24 or 0.9.26 ? Before 0.9.26 the GUI wasn't responsive under load, the only thing you could do was waiting for the CPU load to lower.

Thanks,

luigi

[Forseti]

Luigi, thank you for your help.

For future reference: issue was caused by rabbitmq starting after minemeld had already started.

[lmori]

An additional check for this will be added in the next release.