RabbitMQ not properly starting (failing)

Announcements

Changes to the LIVEcommunity experience are coming soon... Here's what you need to know.

Reply
azuniga
L4 Transporter

RabbitMQ not properly starting (failing)

An issue seen when working with a customer is that rabbitMQ was "exiting" upon startup.

 

service rabbitmq-server stop -- stop the service

service rabbitmq-server start -- start the service

service rabbitmq-server status -- verify rabbitmq is working properly

 

Error code seen critical information found in BOLD

 

expedition@Expedition:~$ service rabbitmq-server status
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2020-04-14 14:07:19 CDT; 380ms ago
  Process: 85685 ExecStop=/bin/sh -c while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
  Process: 85538 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
  Process: 85164 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 85164 (code=exited, status=1/FAILURE)
   Status: "Exited."
 
Apr 14 14:07:19 Expedition systemd[1]: Failed to start RabbitMQ broker.
Apr 14 14:07:19 Expedition systemd[1]: rabbitmq-server.service: Unit entered failed state.
Apr 14 14:07:19 Expedition systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
expedition@Expedition:~$

 

We attempted to restart the server and service multiple times however the service was never able to properly run, so a netstat was run to verify that there is not another service running on the rabbitmq service port and preventing it from running. (Default rabbitmq port = 5672) generally found within the rabbitmq.conf file.

 

Solution:

 

sudo service mysql stop

sudo service rabbitmq-server stop

 

A separate folder was created and we moved the files from ( var/lib/rabbitmq/ ) into that temporary folder after the service was stopped.

 

sudo service mysql start

sudo service rabbitmq-server start

 

We first ran a restart of rabbitmq after moving the files into the temporary folder and verified it worked properly. After this verification we moved it back into the correct path and this resolved the issue.

 

service rabbitmq-server status -- to verify the service is properly running.

 

I believe this should resolve issues seen with rabbitmq, which affects the panordersreaders (Jobs and Task Manager) on the expedition dashboard.

 

Special thanks to @alestevez for his assistance in finding a resolution to this issue!

Tags (1)
StephenW
L1 Bithead

I am also seeing this issue, but trying your suggested solution has not resolved my issue.  After moving the /var/lib/rabbitmq/ files to a temporary directory I get the following output when trying to start rabbitmq:

 

expedition@paexpedition:~$ sudo service rabbitmq-server start
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
expedition@paexpedition:~$ systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2021-03-16 08:59:25 EDT; 3s ago
Process: 3540 ExecStop=/bin/sh -c while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
Process: 3869 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=200/CHDIR)
Process: 3866 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=200/CHDIR)
Main PID: 3866 (code=exited, status=200/CHDIR)

Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=200
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 08:59:25 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
expedition@paexpedition:~$ sudo service rabbitmq-server start
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
expedition@paexpedition:~$ journalctl -xe
Mar 16 09:00:06 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 09:00:06 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Service hold-off time over, scheduling restart.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Stopped RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has finished shutting down.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Starting RabbitMQ broker...
-- Subject: Unit rabbitmq-server.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has begun starting up.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[3967]: rabbitmq-server.service: Failed at step CHDIR spawning /usr/lib/rabbitmq/bin/rabbitmq-server: No such file or direct
-- Subject: Process /usr/lib/rabbitmq/bin/rabbitmq-server could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/lib/rabbitmq/bin/rabbitmq-server could not be executed and failed.
--
-- The error number returned by this process is 2.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=200/CHDIR
Mar 16 09:00:16 paexpedition.wcu.edu systemd[3970]: rabbitmq-server.service: Failed at step CHDIR spawning /usr/lib/rabbitmq/bin/rabbitmqctl: No such file or directory
-- Subject: Process /usr/lib/rabbitmq/bin/rabbitmqctl could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /usr/lib/rabbitmq/bin/rabbitmqctl could not be executed and failed.
--
-- The error number returned by this process is 2.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=200
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has failed.
--
-- The result is failed.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Unit entered failed state.
Mar 16 09:00:16 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

azuniga
L4 Transporter

Hello @StephenW 

 

You should check /var/log/rabbitmq/startup_err

See if there is more information as to why it is failing.

StephenW
L1 Bithead

Thank you for your feedback @azuniga , but that file does not exist on my system.

azuniga
L4 Transporter

Hello @StephenW 

 

Any new content in the folder path "/var/log/rabbitmq/"

 

This is how my folder looks where I store my rabbitmq logs.

Screen Shot 2021-03-16 at 10.43.54 AM.png

StephenW
L1 Bithead

My /var/log/rabbitmq/ looks similar, but I do not have the startup_err and startup_log files.  The last content I see written was the rabbitmq service stopping in rabbit@paexpedition.log.  The journalctl -xe command provides the following information when trying to start the rabbitmq-server service:

-- Unit rabbitmq-server.service has begun starting up.
Mar 16 14:09:50 paexpedition.wcu.edu rabbitmq-server[17940]: mkdir: cannot create directory ‘/var/lib/rabbitmq/mnesia’: Permission denied
Mar 16 14:09:50 paexpedition.wcu.edu rabbitmq-server[17940]: Failed to create directory: /var/lib/rabbitmq/mnesia
Mar 16 14:09:50 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=73/n/a
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: Error: Failed to initialize erlang distribution: {{shutdown,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {failed_to_start_child,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: auth,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {"Failed to create cookie file '/var/lib/rabbitmq/.erlang.c
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{auth,init_cookie,0,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"auth.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,286}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {auth,init,1,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"auth.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,140}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {gen_server,init_it,6,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: "gen_server.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,328}]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {proc_lib,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: init_p_do_apply,3,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [{file,"proc_lib.erl"},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {line,240}]}]}}},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {child,undefined,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: net_sup_dynamic,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: {erl_distribution,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: start_link,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [['rabbitmq-cli-61',
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: shortnames]]},
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: permanent,1000,supervisor,
Mar 16 14:09:51 paexpedition.wcu.edu rabbitmqctl[18017]: [erl_distribution]}}.
Mar 16 14:09:51 paexpedition.wcu.edu systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=75
Mar 16 14:09:51 paexpedition.wcu.edu systemd[1]: Failed to start RabbitMQ broker.

Avanderheyden
L3 Networker

Can you check your hostname(s) and IP mappings set in /etc/hosts are correct?

StephenW
L1 Bithead

The only entries in the /etc/hosts file are the typical 127.0.0.1 pointing to localhost and and entry for the server's IPv4 address pointing both to paexpedition.wcu.edu and paexpedition

azuniga
L4 Transporter

Hello @StephenW 

 

Are you running this as a container? Also it says that the rabbitmq broker is failing to start did you verify everything is functional with rabbitmq? Or it just wont start at all?

StephenW
L1 Bithead

Apologies, I'm not by any means a linux server admin... Expedition is not running as a container, it's a native install on Ubuntu. The RabbitMQ service fails to start at all.


Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!