minemeld-engine is hanging in STARTING state

Reply
L2 Linker

minemeld-engine is hanging in STARTING state

I installed Minemeld on a fresh Ubuntu 16.04.6 LTS with the instructions "Manually Install MineMeld On Ubuntu Server 16.04"  but the minemeld-engine is hanging in STARTING state.  minemeld-engine.log shows a couple of directory not found messages for CheckPoints. Any idea how to fix this?

lab-user@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
[sudo] password for lab-user:
minemeld-engine                  STARTING
minemeld-supervisord-listener    RUNNING   pid 1190, uptime 0:00:19
minemeld-traced                  RUNNING   pid 1192, uptime 0:00:19
minemeld-web                     RUNNING   pid 1193, uptime 0:00:19

redis is running as well

lab-user@minemeld:~$ sudo systemctl status redis
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-07-08 00:44:38 IST; 1h 53min left
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 1069 ExecStartPost=/bin/run-parts --verbose /etc/redis/redis-server.post-up.d (code=exited, status=0/SUCCESS)
  Process: 1032 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
  Process: 1006 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d (code=exited, status=0/SUCCESS)
 Main PID: 1068 (redis-server)
    Tasks: 3
   Memory: 2.8M
      CPU: 589ms
   CGroup: /system.slice/redis-server.service
           └─1068 /usr/bin/redis-server 127.0.0.1:6379

Jul 08 00:44:37 minemeld systemd[1]: Starting Advanced key-value store...
Jul 08 00:44:37 minemeld run-parts[1006]: run-parts: executing /etc/redis/redis-server.pre-up.d/00_example
Jul 08 00:44:38 minemeld systemd[1]: Started Advanced key-value store.
L7 Applicator

Re: minemeld-engine is hanging in STARTING state

@LarsAtConsigas please could you check your redis config is correct:

ubuntu@xenial:~$ grep unix /etc/redis/redis.conf
# on a unix socket when not specified.
unixsocket /var/run/redis/redis.sock
unixsocketperm 770
L2 Linker

Re: minemeld-engine is hanging in STARTING state

Hi Luigi,

 

yes these two entires are in redis.conf

 

Cheers

Lars

L7 Applicator

Re: minemeld-engine is hanging in STARTING state

Could you check the minemeld-engine.log file again for errors? Is the engine still STARTING after 30 seconds?

L2 Linker

Re: minemeld-engine is hanging in STARTING state

I checked the status after the server has been up all day and to my surprise it showed as RUNNING. So I restarted the server again and now after over 20min the status still shows as STARTING

 

I cannot find any obvious errors in minemeld-engine.log but there is a strange timeshift 

2019-07-08T19:02:11 (1210)basepoller._actor_loop INFO: dshield_blocklist - command: 1562608931147 age_out
2019-07-08T19:02:11 (1210)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-08T19:02:11 (1210)basepoller._actor_loop INFO: dshield_blocklist - command: 1562608931147 gc
2019-07-08T19:02:11 (1210)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-07-08T17:03:16 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601796946 age_out
2019-07-08T17:03:16 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command age_out, old timestamp - ignored
2019-07-08T17:04:23 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601863949 age_out
2019-07-08T17:04:23 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command age_out, old timestamp - ignored
2019-07-08T17:05:30 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601930950 age_out

and the other processes show a negative uptime

 

lab-user@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine                  STARTING
minemeld-supervisord-listener    RUNNING   pid 1195, uptime -1 day, 22:15:41
minemeld-traced                  RUNNING   pid 1197, uptime -1 day, 22:15:41
minemeld-web                     RUNNING   pid 1198, uptime -1 day, 22:15:41

 

L2 Linker

Re: minemeld-engine is hanging in STARTING state

After a couple of hours the minemeld-engine process shows as RUNNING. Any idea what is causing this long delay?

L7 Applicator

Re: minemeld-engine is hanging in STARTING state

@LarsAtConsigas it seems you have a problem with the clock/ntp settings of your instance. This is not related to MineMeld. Please double check your system logs and system config for clock updates.

L2 Linker

Re: minemeld-engine is hanging in STARTING state

Thanks Luigi, turns out that the standard Ubuntu install didn't link the hardware clock with the system clock and the hardware clock was 2h in the future. The minemeld engine now starts imidiately after fixing this.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!