rabbitmq-server fails to start on Red Hat Openstack Platform 7

Solution Verified - Updated -

Environment

Red Hat Openstack Platform 7

Issue

Following a server outage, the rabbitmq-server service fails to start.

Resolution

In /var/lib/rabbitmq a dump file named erl_crash.dump will be produced.

In that same directory there exists a directory named mnesia.

Remove the contents of the mnesia directory and restart the rabbitmq-server service. For example:

# rm -rf /var/lib/rabbitmq/mnesia/*

# systemctl start rabbitmq-server

Depending on your configuration it is likely that other services did not start correctly because they depend on rabbitmq-server. Running openstack-status will reveal the state of active, inactive and failed services. Those can be restarted using systemctl as illustrated above.

Root Cause

In the case of an abrupt server outage such as a crash or hardware event, the state files in the /var/lib/rabbitmq/mnesia directory can become corrupted and unusable, causing rabbitmq-server to fail to start. Removing the corrupted files allows the service to start normally.

Diagnostic Steps

Examine the status of the service using the systemctl command as follows:

[root@openstack3 ~]# systemctl status rabbitmq-server.service -l
rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Mon 2015-08-31 19:03:02 EDT; 47s ago
  Process: 3981 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=2)
  Process: 1514 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
 Main PID: 1514 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/rabbitmq-server.service

Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: * epmd reports: node 'rabbit' not running at all
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: no other nodes on openstack3
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: * suggestion: start the node
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: current node details:
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: - node name: rabbitmqctl3981@openstack3
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: - home dir: /var/lib/rabbitmq
Aug 31 19:03:02 openstack3 rabbitmqctl[3981]: - cookie hash: QYtpShhTvu3TyQQuFtgvVw==
Aug 31 19:03:02 openstack3 systemd[1]: rabbitmq-server.service: control process exited, code=exited status=2
Aug 31 19:03:02 openstack3 systemd[1]: Failed to start RabbitMQ broker.
Aug 31 19:03:02 openstack3 systemd[1]: Unit rabbitmq-server.service entered failed state.

Run openstack-status and observe that the rabbitmq-server service is listed as failed.

== Support services ==
mysqld:                                 inactive  (disabled on boot)
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
target:                                 active
rabbitmq-server:                        failed
memcached:                              active

Examine the log file at /var/log/rabbitmq/rabbit@hostname.log for lines similar to the following:


=INFO REPORT==== 31-Aug-2015::18:40:52 === Error description: {error,corrupt_cluster_status_files,[3]} Log files (may contain more information): /var/log/rabbitmq/rabbit@openstack3.log /var/log/rabbitmq/rabbit@openstack3-sasl.log Stack trace: [{rabbit_node_monitor,'-prepare_cluster_status_files/0-fun-0-',1, [{file,"src/rabbit_node_monitor.erl"},{line,100}]}, {rabbit_node_monitor,prepare_cluster_status_files,0, [{file,"src/rabbit_node_monitor.erl"},{line,103}]}, {rabbit,'-boot/0-fun-1-',0,[{file,"src/rabbit.erl"},{line,328}]}, {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,358}]}, {init,start_it,1,[]}, {init,start_em,1,[]}]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments