topic Re: Error reading last checkpoint in General Topics

Error reading last checkpoint

Forseti — Tue, 15 Nov 2016 08:20:02 GMT

Hi everyone,

I am facing an issue that floods my output SIEM a little to often. The issue seems to be that the miner node is unable to register where it left of during the last check. Any tips on solving this?

2016-11-14T08:54:30 (17269)base.read_checkpoint ERROR: sslabusech_dyreblacklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 245, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'sslabusech_dyreblacklist.chkp'
2016-11-14T08:54:30 (17269)base.state INFO: sslabusech_dyreblacklist - transitioning to state 1
2016-11-14T08:54:30 (17270)base.read_checkpoint ERROR: sslabusech_ipblacklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 245, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'sslabusech_ipblacklist.chkp'
2016-11-14T08:54:30 (17270)base.state INFO: sslabusech_ipblacklist - transitioning to state 1
2016-11-14T08:54:30 (17270)base.read_checkpoint ERROR: openbl_base - Error reading last checkpoint

Regards,

Forseti

Re: Error reading last checkpoint

lmori — Tue, 15 Nov 2016 08:28:01 GMT

Hi @Forseti,

that ERROR happens only when the engine starts if the checkpoint does not exist. It could happen for 2 reasons:

- the node is new and there is no previous checkpoint to load

- the engine didn't have a "clean" shut down

- could you check before those ERROR messages if you see additional ERRORs ?

- check also the OS syslog and dmesg to see if there was a problem with memory or disk exhaustion

Thanks,

luigi

Re: Error reading last checkpoint

Forseti — Tue, 15 Nov 2016 08:54:09 GMT

Hi Imori, The machine gets rebooted at night because as it seems that the minemeld process isn't stable. (I wasn't able to logon via the web gui after a couple of hours) I can't seem to stop it properly. Neither using 'sudo service mineld stop' nor via supervisord. I've already removed and reinstalled MineMeld completely but the issue seems to persist. However, that seems to be the root of this issue. And ofcourse, now that I try to mimick the behavior, it all goes well.. Will check back when a get a sample! Regards, Jan

Re: Error reading last checkpoint

lmori — Tue, 15 Nov 2016 09:10:01 GMT

I'd like to fix the instability issue first, I think it's related to memory exhaustion.

Would you mind unicast me the logs from /opt/minemeld/log ? My email is lmori@paloaltonetworks.com.

Also please note that when MineMeld start without a checkpoint it starts crunching all the indicators, and it could take a while. Are your running 0.9.24 or 0.9.26 ? Before 0.9.26 the GUI wasn't responsive under load, the only thing you could do was waiting for the CPU load to lower.

Thanks,

luigi

Re: Error reading last checkpoint

Forseti — Tue, 15 Nov 2016 12:56:09 GMT

Luigi, thank you for your help.

For future reference: issue was caused by rabbitmq starting after minemeld had already started.

Re: Error reading last checkpoint

lmori — Wed, 16 Nov 2016 07:23:01 GMT

An additional check for this will be added in the next release.