Temporary loss of quorum in RHEL 6 cluster when using master_wins and gracefully shutting down the master node

Solution Verified - Updated -

Issue

  • In a two-node cluster setup with a quorum device, the quorum temporarily dissolves when stopping the qdisk master node.
Apr 18 17:23:50 jrummy6-1-clust qdiskd[13247]: Node 2 shutdown
Apr 18 17:23:50 jrummy6-1-clust kernel: dlm: closing connection to node 2
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [CMAN  ] quorum lost, blocking activity
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [QUORUM] Members[1]: 1
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [CPG   ] chosen downlist: sender r(0) ip(192.168.143.61) ; members(old:2 left:1)
Apr 18 17:23:50 jrummy6-1-clust corosync[13201]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 17:23:51 jrummy6-1-clust corosync[13201]:   [CMAN  ] quorum device re-registered
Apr 18 17:23:51 jrummy6-1-clust corosync[13201]:   [CMAN  ] quorum regained, resuming activity
Apr 18 17:23:51 jrummy6-1-clust corosync[13201]:   [QUORUM] This node is within the primary component and will provide service.
Apr 18 17:23:51 jrummy6-1-clust corosync[13201]:   [QUORUM] Members[1]: 1
Apr 18 17:23:51 jrummy6-1-clust qdiskd[13247]: Assuming master role
  • When the qdisk master leaves in a master_wins configuration, quorum is lost on the remaining node and services are disabled rather than relocating to it.
  • Cannot fail over service when shutting down a node

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
  • cluster (cman, clusterlib) release prior to 3.0.12.1-49.el6
  • Use of a quorum disk (qdisk)
  • One of the following:
    • master_wins="1" set in the quorumd tag in /etc/cluster/cluster.conf
    • A 2-node cluster with no heuristics defined

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.