topic Re: minemeld-engine is hanging in STARTING state in General Topics

minemeld-engine is hanging in STARTING state

LarsAtConsigas — Sun, 07 Jul 2019 21:56:55 GMT

I installed Minemeld on a fresh Ubuntu 16.04.6 LTS with the instructions "Manually Install MineMeld On Ubuntu Server 16.04" but the minemeld-engine is hanging in STARTING state. minemeld-engine.log shows a couple of directory not found messages for CheckPoints. Any idea how to fix this?

lab-user@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
[sudo] password for lab-user:
minemeld-engine                  STARTING
minemeld-supervisord-listener    RUNNING   pid 1190, uptime 0:00:19
minemeld-traced                  RUNNING   pid 1192, uptime 0:00:19
minemeld-web                     RUNNING   pid 1193, uptime 0:00:19

redis is running as well

lab-user@minemeld:~$ sudo systemctl status redis
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-07-08 00:44:38 IST; 1h 53min left
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 1069 ExecStartPost=/bin/run-parts --verbose /etc/redis/redis-server.post-up.d (code=exited, status=0/SUCCESS)
  Process: 1032 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
  Process: 1006 ExecStartPre=/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d (code=exited, status=0/SUCCESS)
 Main PID: 1068 (redis-server)
    Tasks: 3
   Memory: 2.8M
      CPU: 589ms
   CGroup: /system.slice/redis-server.service
           └─1068 /usr/bin/redis-server 127.0.0.1:6379

Jul 08 00:44:37 minemeld systemd[1]: Starting Advanced key-value store...
Jul 08 00:44:37 minemeld run-parts[1006]: run-parts: executing /etc/redis/redis-server.pre-up.d/00_example
Jul 08 00:44:38 minemeld systemd[1]: Started Advanced key-value store.

Re: minemeld-engine is hanging in STARTING state

lmori — Mon, 08 Jul 2019 13:24:37 GMT

@LarsAtConsigas please could you check your redis config is correct:

ubuntu@xenial:~$ grep unix /etc/redis/redis.conf
# on a unix socket when not specified.
unixsocket /var/run/redis/redis.sock
unixsocketperm 770

Re: minemeld-engine is hanging in STARTING state

LarsAtConsigas — Mon, 08 Jul 2019 15:53:24 GMT

Hi Luigi,

yes these two entires are in redis.conf

Cheers

Lars

Re: minemeld-engine is hanging in STARTING state

lmori — Mon, 08 Jul 2019 15:57:44 GMT

Could you check the minemeld-engine.log file again for errors? Is the engine still STARTING after 30 seconds?

Re: minemeld-engine is hanging in STARTING state

LarsAtConsigas — Mon, 08 Jul 2019 16:22:43 GMT

I checked the status after the server has been up all day and to my surprise it showed as RUNNING. So I restarted the server again and now after over 20min the status still shows as STARTING

I cannot find any obvious errors in minemeld-engine.log but there is a strange timeshift

2019-07-08T19:02:11 (1210)basepoller._actor_loop INFO: dshield_blocklist - command: 1562608931147 age_out
2019-07-08T19:02:11 (1210)table._query_by_index INFO: Deleted in scan of _age_out: 0
2019-07-08T19:02:11 (1210)basepoller._actor_loop INFO: dshield_blocklist - command: 1562608931147 gc
2019-07-08T19:02:11 (1210)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2019-07-08T17:03:16 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601796946 age_out
2019-07-08T17:03:16 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command age_out, old timestamp - ignored
2019-07-08T17:04:23 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601863949 age_out
2019-07-08T17:04:23 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command age_out, old timestamp - ignored
2019-07-08T17:05:30 (1210)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1562601930950 age_out

and the other processes show a negative uptime

lab-user@minemeld:~$ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status
minemeld-engine                  STARTING
minemeld-supervisord-listener    RUNNING   pid 1195, uptime -1 day, 22:15:41
minemeld-traced                  RUNNING   pid 1197, uptime -1 day, 22:15:41
minemeld-web                     RUNNING   pid 1198, uptime -1 day, 22:15:41

Re: minemeld-engine is hanging in STARTING state

LarsAtConsigas — Tue, 09 Jul 2019 06:34:25 GMT

After a couple of hours the minemeld-engine process shows as RUNNING. Any idea what is causing this long delay?

Re: minemeld-engine is hanging in STARTING state

lmori — Tue, 09 Jul 2019 07:32:21 GMT

@LarsAtConsigas it seems you have a problem with the clock/ntp settings of your instance. This is not related to MineMeld. Please double check your system logs and system config for clock updates.

Re: minemeld-engine is hanging in STARTING state

LarsAtConsigas — Wed, 10 Jul 2019 07:07:38 GMT

Thanks Luigi, turns out that the standard Ubuntu install didn't link the hardware clock with the system clock and the hardware clock was 2h in the future. The minemeld engine now starts imidiately after fixing this.