topic Minemeld crash once in a while in General Topics

Minemeld crash once in a while

yguo29 — Mon, 11 May 2020 17:40:15 GMT

HI,

My Minemeld is running in a docker container. It consumes memory as much as possible. Now the server is configured with 32 GB memory, still 100% usage by Minemeld. It crashes every a couple of days. The requested url for feed gets internal error. I have to restart the container to fix it.

We are processing lots of indicators. Total number is 2 million.

Any idea how to properly configure the app or suggestion for sizing?

Thank you.

Re: Minemeld crash once in a while a

DanieleDenisi — Mon, 25 May 2020 17:23:49 GMT

I had the same issue on a ubuntu 18.04 installation, with minemeld installed by package ( following this guide Manually Install MineMeld on Ubuntu 16.04 ).

On dmesg logs I foud a segfault crash :

[629369.655024] mm-traced[13626]: segfault at 2ce5 ip 00007f70920e613d sp 00007ffd009b6cb8 error 4 in libleveldb.so.1.20[7f70920c3000+5a000]
[629370.822671] mm-traced[13680]: segfault at 2ce5 ip 00007fb8c887113d sp 00007ffd4fcab598 error 4 in libleveldb.so.1.20[7fb8c884e000+5a000]
[629419.269495] mm-run[13690]: segfault at 655e ip 00007f5fddd7213d sp 00007f5fd76fd828 error 4 in libleveldb.so.1.20[7f5fddd4f000+5a000]
[629420.710727] mm-traced[13685]: segfault at 2ce5 ip 00007f895010e13d sp 00007ffeec3e3868 error 4 in libleveldb.so.1.20[7f89500eb000+5a000]
[631226.662338] mm-traced[13693]: segfault at 2ce5 ip 00007f161f92113d sp 00007fff42889188 error 4 in libleveldb.so.1.20[7f161f8fe000+5a000]
[631845.671715] mm-traced[13706]: segfault at 2ce5 ip 00007f5728ef113d sp 00007ffd2292dd08 error 4 in libleveldb.so.1.20[7f5728ece000+5a000]
[649628.510422] mm-traced[13712]: segfault at 200000 ip 00007ff429dbe31a sp 00007ffd953acb00 error 4 in _plyvel.so[7ff429da9000+27000]

Googling I have found that someone had the same issue and solved downgrading libleveldb to version 1.18.

I tried it and I solved the problem with the steps below :

download libleveldb v.1.18 from official repo https://github.com/google/leveldb/releases/tag/v1.18 ;
compile libleveldb.so.1.18 and copy it to /usr/local/lib : tar xzvf leveldb-1.18.tar.gz

cd leveldb-1.18

make

cp libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/

cp -r leveldb /usr/local/include
At this point I updated all the symlink to point to libleveldb.so.1.18 instead of libleveldb.so.1.20, but after performing the ldconfig command , they changed back to libleveldb.so.1.20 .So, I did a bad thing :

cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20 /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20.backup

cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.18 usr/lib/x86_64-linux-gnu/libleveldb.so.1.20

I know it is an inelegant workaround but after a restart of the service ( better a reboot ) everything works fine and minemeld did not crash any more .

Maybe the container you are using has been updated and some libraries conflicts with other libraries inside minemeld .

In my case the conflict was betweend these 2 :

/usr/lib/x86_64-linux-gnu/libleveldb.so.1.20
/opt/minemeld/engine/0.9.70/lib/python2.7/site-packages/plyvel/_plyvel.so

Ciao,

Daniele

Re: Minemeld crash once in a while

DanieleDenisi — Wed, 27 May 2020 09:13:18 GMT

Hi,

I had the same problem in an installation of minemeld on ubuntu 18.04 ( from package following this guide ).

Looking on dmesg logs I found the errors below:

[627512.663356] mm-traced[13568]: segfault at 2ce5 ip 00007f52c4bba13d sp 00007ffe0a5d4278 error 4 in libleveldb.so.1.20[7f52c4b97000+5a000] [627514.503022] mm-traced[13621]: segfault at 2ce5 ip 00007f5df770713d sp 00007ffe423cfd18 error 4 in libleveldb.so.1.20[7f5df76e4000+5a000] [629369.655024] mm-traced[13626]: segfault at 2ce5 ip 00007f70920e613d sp 00007ffd009b6cb8 error 4 in libleveldb.so.1.20[7f70920c3000+5a000] [629370.822671] mm-traced[13680]: segfault at 2ce5 ip 00007fb8c887113d sp 00007ffd4fcab598 error 4 in libleveldb.so.1.20[7fb8c884e000+5a000] [629419.269495] mm-run[13690]: segfault at 655e ip 00007f5fddd7213d sp 00007f5fd76fd828 error 4 in libleveldb.so.1.20[7f5fddd4f000+5a000] [629420.710727] mm-traced[13685]: segfault at 2ce5 ip 00007f895010e13d sp 00007ffeec3e3868 error 4 in libleveldb.so.1.20[7f89500eb000+5a000] [631226.662338] mm-traced[13693]: segfault at 2ce5 ip 00007f161f92113d sp 00007fff42889188 error 4 in libleveldb.so.1.20[7f161f8fe000+5a000] [631845.671715] mm-traced[13706]: segfault at 2ce5 ip 00007f5728ef113d sp 00007ffd2292dd08 error 4 in libleveldb.so.1.20[7f5728ece000+5a000] [649628.510422] mm-traced[13712]: segfault at 200000 ip 00007ff429dbe31a sp 00007ffd953acb00 error 4 in _plyvel.so[7ff429da9000+27000] [660212.351922] SGI XFS with ACLs, security attributes, realtime, no debug enabled

Googling I found that someone had the same problem caused by an incompatibility between libleveldb.so.1.20 and _plyvel.so and

downgrading to libdevel to 1.18 solved the problem.

I tried it with success and now minemeld works smootly.

I foolwed the steps below :

download libdevel 1.18 from https://github.com/google/leveldb/releases/tag/v1.18
cd leveldb-1.18
make
cp libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/

At this point I updated sym link to point to libleveldb.so.1.18 instead of libleveldb.so.1.20 and issued ldconfig command.

But after this , the symlink pointed back to libdevel.so.1.20.

So I did a bad thing :

cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20.

It is a very inelegant work around but It solved.

After a restart of minemeld , better a reboot, now everything works fine.