topic Re: RabbitMQ not properly starting (failing) in Expedition Discussions

RabbitMQ not properly starting (failing)

azuniga — Wed, 15 Apr 2020 04:42:15 GMT

An issue seen when working with a customer is that rabbitMQ was "exiting" upon startup.

service rabbitmq-server stop -- stop the service

service rabbitmq-server start -- start the service

service rabbitmq-server status -- verify rabbitmq is working properly

Error code seen critical information found in BOLD

expedition@Expedition:~$ service rabbitmq-server status
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2020-04-14 14:07:19 CDT; 380ms ago
Process: 85685 ExecStop=/bin/sh -c while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
Process: 85538 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 85164 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
Main PID: 85164 (code=exited, status=1/FAILURE)
Status: "Exited."

Apr 14 14:07:19 Expedition systemd[1]: Failed to start RabbitMQ broker.
Apr 14 14:07:19 Expedition systemd[1]: rabbitmq-server.service: Unit entered failed state.
Apr 14 14:07:19 Expedition systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
expedition@Expedition:~$

We attempted to restart the server and service multiple times however the service was never able to properly run, so a netstat was run to verify that there is not another service running on the rabbitmq service port and preventing it from running. (Default rabbitmq port = 5672) generally found within the rabbitmq.conf file.

Solution:

sudo service mysql stop

sudo service rabbitmq-server stop

A separate folder was created and we moved the files from ( var/lib/rabbitmq/ ) into that temporary folder after the service was stopped.

sudo service mysql start

sudo service rabbitmq-server start

We first ran a restart of rabbitmq after moving the files into the temporary folder and verified it worked properly. After this verification we moved it back into the correct path and this resolved the issue.

service rabbitmq-server status -- to verify the service is properly running.

I believe this should resolve issues seen with rabbitmq, which affects the panordersreaders (Jobs and Task Manager) on the expedition dashboard.

Special thanks to @alestevez for his assistance in finding a resolution to this issue!

Re: RabbitMQ not properly starting (failing)

StephenW — Tue, 16 Mar 2021 13:14:47 GMT

I am also seeing this issue, but trying your suggested solution has not resolved my issue. After moving the /var/lib/rabbitmq/ files to a temporary directory I get the following output when trying to start rabbitmq:

expedition@paexpedition:~$ sudo service rabbitmq-server start
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
expedition@paexpedition:~$ systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2021-03-16 08:59:25 EDT; 3s ago
Process: 3540 ExecStop=/bin/sh -c while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
Process: 3869 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=200/CHDIR)
Process: 3866 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=200/CHDIR)
Main PID: 3866 (code=exited, status=200/CHDIR)

Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=200
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
expedition@paexpedition:~$ sudo service rabbitmq-server start
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
expedition@paexpedition:~$ journalctl -xe
Mar 16 09:00:06 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 09:00:06 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Service hold-off time over, scheduling restart.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Stopped RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has finished shutting down.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Starting RabbitMQ broker...
-- Subject: Unit rabbitmq-server.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has begun starting up.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[3967]: rabbitmq-server.service: Failed at step CHDIR spawning /usr/lib/rabbitmq/bin/rabbitmq-server: No such file or direct
-- Subject: Process /usr/lib/rabbitmq/bin/rabbitmq-server could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/lib/rabbitmq/bin/rabbitmq-server could not be executed and failed.
--
-- The error number returned by this process is 2.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=200/CHDIR
Mar 16 09:00:16 paexpedition.wcu.edu systemd[3970]: rabbitmq-server.service: Failed at step CHDIR spawning /usr/lib/rabbitmq/bin/rabbitmqctl: No such file or directory
-- Subject: Process /usr/lib/rabbitmq/bin/rabbitmqctl could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/lib/rabbitmq/bin/rabbitmqctl could not be executed and failed.
--
-- The error number returned by this process is 2.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=200
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has failed.
--
-- The result is failed.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

Re: RabbitMQ not properly starting (failing)

azuniga — Tue, 16 Mar 2021 15:26:24 GMT

Hello @StephenW

You should check /var/log/rabbitmq/startup_err

See if there is more information as to why it is failing.

Re: RabbitMQ not properly starting (failing)

StephenW — Tue, 16 Mar 2021 17:26:06 GMT

Thank you for your feedback @azuniga , but that file does not exist on my system.

Re: RabbitMQ not properly starting (failing)

azuniga — Tue, 16 Mar 2021 17:45:25 GMT

Hello @StephenW

Any new content in the folder path "/var/log/rabbitmq/"

This is how my folder looks where I store my rabbitmq logs.

Re: RabbitMQ not properly starting (failing)

StephenW — Tue, 16 Mar 2021 18:14:38 GMT

My /var/log/rabbitmq/ looks similar, but I do not have the startup_err and startup_log files. The last content I see written was the rabbitmq service stopping in rabbit@paexpedition.log. The journalctl -xe command provides the following information when trying to start the rabbitmq-server service:

-- Unit rabbitmq-server.service has begun starting up.
Mar 16 14:09:50 paexpedition.wcu.edu rabbitmq-server[17940]: mkdir: cannot create directory ‘/var/lib/rabbitmq/mnesia’: Permission denied
Mar 16 14:09:50 paexpedition.wcu.edu rabbitmq-server[17940]: Failed to create directory: /var/lib/rabbitmq/mnesia
Mar 16 14:09:50 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=73/n/a
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: Error: Failed to initialize erlang distribution: {{shutdown,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {failed_to_start_child,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: auth,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {"Failed to create cookie file '/var/lib/rabbitmq/.erlang.c
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{auth,init_cookie,0,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"auth.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,286}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {auth,init,1,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"auth.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,140}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {gen_server,init_it,6,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: "gen_server.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,328}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {proc_lib,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: init_p_do_apply,3,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"proc_lib.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,240}]}]}}},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {child,undefined,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: net_sup_dynamic,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {erl_distribution,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: start_link,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [['rabbitmq-cli-61',
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: shortnames]]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: permanent,1000,supervisor,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [erl_distribution]}}.
Mar 16 14:09:51 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=75
Mar 16 14:09:51 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.

Re: RabbitMQ not properly starting (failing)

Arne-VDH — Mon, 22 Mar 2021 15:37:36 GMT

Can you check your hostname(s) and IP mappings set in /etc/hosts are correct?

Re: RabbitMQ not properly starting (failing)

StephenW — Tue, 23 Mar 2021 12:23:56 GMT

The only entries in the /etc/hosts file are the typical 127.0.0.1 pointing to localhost and and entry for the server's IPv4 address pointing both to paexpedition.wcu.edu and paexpedition

Re: RabbitMQ not properly starting (failing)

azuniga — Tue, 23 Mar 2021 15:57:19 GMT

Hello @StephenW

Are you running this as a container? Also it says that the rabbitmq broker is failing to start did you verify everything is functional with rabbitmq? Or it just wont start at all?

Re: RabbitMQ not properly starting (failing)

StephenW — Fri, 26 Mar 2021 12:22:03 GMT

Apologies, I'm not by any means a linux server admin... Expedition is not running as a container, it's a native install on Ubuntu. The RabbitMQ service fails to start at all.

Re: RabbitMQ not properly starting (failing)

azuniga — Fri, 26 Mar 2021 15:23:16 GMT

Hello @StephenW

Have you tried the following command?

sudo service rabbitmq-server start

Re: RabbitMQ not properly starting (failing)

StephenW — Fri, 26 Mar 2021 16:58:07 GMT

I have, it fails to start. I posted as much information as I know how to gather ("systemctl status rabbitmq-server.service" and "journalctl -xe") in an earlier post.