cman fails to start with "Timed-out waiting for cluster" and openais repeatedly logs "The consensus timeout expired." immediately after starting cman in RHEL 5

Solution Unverified - Updated -

Issue

  • When cman starts either automatically on boot or by manually running service cman start, it reports "Timed-out waiting for cluster"
# service cman start
Starting cluster: 
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... failed
Timed-out waiting for cluster
                                                           [FAILED]
  • In /var/log/messages I see repeated reports of The consensus timeout expired and entering GATHER state from 3:
Apr 12 17:06:45  openais[14610]: [MAIN ] Service initialized 'openais CMAN membership service 2.01'
Apr 12 17:06:45  openais[14610]: [SERV ] Service initialized 'openais extended virtual synchrony service'
Apr 12 17:06:45  openais[14610]: [SERV ] Service initialized 'openais cluster membership service B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais availability management framework B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais checkpoint service B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais event service B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais distributed locking service B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais message service B.01.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais configuration service'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais cluster closed process group service v1.01'
Apr 12 17:06:46  openais[14610]: [SERV ] Service initialized 'openais cluster config database access v1.01'
Apr 12 17:06:46  openais[14610]: [SYNC ] Not using a virtual synchrony filter.
Apr 12 17:06:46  openais[14610]: [MAIN ] Publishing socket for client connections.
Apr 12 17:06:50  ccsd[14604]: Cluster is not quorate.  Refusing connection.
Apr 12 17:06:50  ccsd[14604]: Error while processing connect: Connection refused 
Apr 12 17:06:55  ccsd[14604]: Cluster is not quorate.  Refusing connection.
Apr 12 17:06:55  ccsd[14604]: Error while processing connect: Connection refused 
Apr 12 17:06:57  openais[14610]: [TOTEM] The consensus timeout expired.
Apr 12 17:06:57  openais[14610]: [TOTEM] entering GATHER state from 3.
  • When I restart the cman service then it is stopping at fencing

Environment

  • Red Hat Enterprise Linux (RHEL) 5 with the High Availability Add On
  • A multicast element in /etc/cluster/cluster.conf that specifies an invalid multicast address (one outside the 224.0.0.1 - 239.255.255.255 range), such as:
<cman expected_votes="1" two_node="1".
   <multicast addr="192.168.2.10"/>
</cman>

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.