cman service start failed with "Starting cman... Timed-out waiting for cluster" on a RHEL 6 High Availability cluster node in a cluster with more than two nodes

Solution Verified - Updated -

Issue

  • We reboot all three nodes regularly, and sometimes one of the nodes comes up without rgmanager started after seeing "Waiting for quorum"... Timed-out waiting for cluster"

    Starting cluster: 
       Checking if cluster has been disabled at boot... [  OK  ]
       Checking Network Manager... [  OK  ]
       Global setup... [  OK  ]
       Loading kernel modules... [  OK  ]
       Mounting configfs... [  OK  ]
       Starting cman... [  OK  ]
       Waiting for quorum... Timed-out waiting for cluster
    [FAILED]
    
  • After a reboot, we found the 3rd node had failed to start cman and join the cluster

    # service cman status
    fenced is stopped
    # service rgmanager status
    rgmanager dead but pid file exists
    

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
  • Cluster of more than two nodes
  • cman is chkconfig'd on at boot

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.