Minemeld crash once in a while

[yguo29]

HI,

My Minemeld is running in a docker container. It consumes memory as much as possible.  Now the server is configured with 32 GB memory, still 100% usage by Minemeld. It crashes every a couple of days. The requested url for feed gets internal error. I have to restart the container to fix it.

We are processing lots of indicators. Total number is 2 million.

Any idea how to properly configure the app or suggestion for sizing?

Thank you.

[DanieleDenisi]

I had the same issue on  a ubuntu 18.04 installation, with minemeld installed by package ( following this guide  Manually Install MineMeld on Ubuntu 16.04  ).

On dmesg logs I foud  a segfault crash :

Spoiler

[629369.655024] mm-traced[13626]: segfault at 2ce5 ip 00007f70920e613d sp 00007ffd009b6cb8 error 4 in libleveldb.so.1.20[7f70920c3000+5a000]
[629370.822671] mm-traced[13680]: segfault at 2ce5 ip 00007fb8c887113d sp 00007ffd4fcab598 error 4 in libleveldb.so.1.20[7fb8c884e000+5a000]
[629419.269495] mm-run[13690]: segfault at 655e ip 00007f5fddd7213d sp 00007f5fd76fd828 error 4 in libleveldb.so.1.20[7f5fddd4f000+5a000]
[629420.710727] mm-traced[13685]: segfault at 2ce5 ip 00007f895010e13d sp 00007ffeec3e3868 error 4 in libleveldb.so.1.20[7f89500eb000+5a000]
[631226.662338] mm-traced[13693]: segfault at 2ce5 ip 00007f161f92113d sp 00007fff42889188 error 4 in libleveldb.so.1.20[7f161f8fe000+5a000]
[631845.671715] mm-traced[13706]: segfault at 2ce5 ip 00007f5728ef113d sp 00007ffd2292dd08 error 4 in libleveldb.so.1.20[7f5728ece000+5a000]
[649628.510422] mm-traced[13712]: segfault at 200000 ip 00007ff429dbe31a sp 00007ffd953acb00 error 4 in _plyvel.so[7ff429da9000+27000]

Googling I have found that someone had the same issue and solved downgrading libleveldb to version 1.18.

I tried it and I solved the problem  with the steps below :

• download libleveldb v.1.18 from  official repo  https://github.com/google/leveldb/releases/tag/v1.18   ;
• compile libleveldb.so.1.18 and copy it to /usr/local/lib  :
  Spoiler

  tar xzvf leveldb-1.18.tar.gz
  
  cd leveldb-1.18
  
  make
  
   cp libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/
  
  cp -r leveldb /usr/local/include
  
  
  
• At this point I updated all the symlink to point to libleveldb.so.1.18 instead of libleveldb.so.1.20, but after performing the ldconfig command , they changed back to libleveldb.so.1.20 .So, I did a bad thing  :

Spoiler

cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20 /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20.backup

cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.18 usr/lib/x86_64-linux-gnu/libleveldb.so.1.20

I know it is an inelegant workaround but after a restart of the service ( better a reboot ) everything works fine and minemeld did not crash any more .

Maybe the container you are using has been updated and some libraries conflicts with other libraries inside minemeld .

In my case the conflict was betweend these 2 :

Spoiler

/usr/lib/x86_64-linux-gnu/libleveldb.so.1.20
/opt/minemeld/engine/0.9.70/lib/python2.7/site-packages/plyvel/_plyvel.so

Ciao,

Daniele

[DanieleDenisi]

Hi,

I had the same problem in an installation of minemeld on ubuntu 18.04 ( from package following  this guide  ).

Looking on dmesg logs I found the  errors below:

[627512.663356] mm-traced[13568]: segfault at 2ce5 ip 00007f52c4bba13d sp 00007ffe0a5d4278 error 4 in libleveldb.so.1.20[7f52c4b97000+5a000]
[627514.503022] mm-traced[13621]: segfault at 2ce5 ip 00007f5df770713d sp 00007ffe423cfd18 error 4 in libleveldb.so.1.20[7f5df76e4000+5a000]
[629369.655024] mm-traced[13626]: segfault at 2ce5 ip 00007f70920e613d sp 00007ffd009b6cb8 error 4 in libleveldb.so.1.20[7f70920c3000+5a000]
[629370.822671] mm-traced[13680]: segfault at 2ce5 ip 00007fb8c887113d sp 00007ffd4fcab598 error 4 in libleveldb.so.1.20[7fb8c884e000+5a000]
[629419.269495] mm-run[13690]: segfault at 655e ip 00007f5fddd7213d sp 00007f5fd76fd828 error 4 in libleveldb.so.1.20[7f5fddd4f000+5a000]
[629420.710727] mm-traced[13685]: segfault at 2ce5 ip 00007f895010e13d sp 00007ffeec3e3868 error 4 in libleveldb.so.1.20[7f89500eb000+5a000]
[631226.662338] mm-traced[13693]: segfault at 2ce5 ip 00007f161f92113d sp 00007fff42889188 error 4 in libleveldb.so.1.20[7f161f8fe000+5a000]
[631845.671715] mm-traced[13706]: segfault at 2ce5 ip 00007f5728ef113d sp 00007ffd2292dd08 error 4 in libleveldb.so.1.20[7f5728ece000+5a000]
[649628.510422] mm-traced[13712]: segfault at 200000 ip 00007ff429dbe31a sp 00007ffd953acb00 error 4 in _plyvel.so[7ff429da9000+27000]
[660212.351922] SGI XFS with ACLs, security attributes, realtime, no debug enabled

Googling I found that someone had the same problem caused by an incompatibility between libleveldb.so.1.20 and _plyvel.so  and

downgrading to  libdevel to 1.18  solved the problem.

I tried it with success and now minemeld works smootly.

I foolwed the steps below :

• download libdevel 1.18 from https://github.com/google/leveldb/releases/tag/v1.18
• cd leveldb-1.18
• make
• cp libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/

At this point I updated sym link to point to libleveldb.so.1.18 instead of libleveldb.so.1.20 and issued  ldconfig command.

But after  this , the symlink pointed back to libdevel.so.1.20. 

So I did a bad thing :

  cp /usr/lib/x86_64-linux-gnu/libleveldb.so.1.18 /usr/lib/x86_64-linux-gnu/libleveldb.so.1.20.

It is a very inelegant work around but It solved.

After a restart of minemeld , better a reboot, now everything works fine.