Temporary loss of quorum in RHEL 6 cluster when using master_wins and gracefully shutting down the master node

Solution Verified - Updated -

Issue

  • In a two-node cluster setup with a quorum device, the quorum temporarily dissolves when stopping the qdisk master node.
Apr 18 17:23:50 hostname1 qdiskd[13247]: Node 2 shutdown
Apr 18 17:23:50 hostname1 kernel: dlm: closing connection to node 2
Apr 18 17:23:50 hostname1 corosync[13201]:   [CMAN  ] quorum lost, blocking activity
Apr 18 17:23:50 hostname1 corosync[13201]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Apr 18 17:23:50 hostname1 corosync[13201]:   [QUORUM] Members[1]: 1
Apr 18 17:23:50 hostname1 corosync[13201]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Apr 18 17:23:50 hostname1 corosync[13201]:   [CPG   ] chosen downlist: sender r(0) ip(X.X.X.X) ; members(old:2 left:1)
Apr 18 17:23:50 hostname1 corosync[13201]:   [MAIN  ] Completed service synchronization, ready to provide service.
Apr 18 17:23:51 hostname1 corosync[13201]:   [CMAN  ] quorum device re-registered
Apr 18 17:23:51 hostname1 corosync[13201]:   [CMAN  ] quorum regained, resuming activity
Apr 18 17:23:51 hostname1 corosync[13201]:   [QUORUM] This node is within the primary component and will provide service.
Apr 18 17:23:51 hostname1 corosync[13201]:   [QUORUM] Members[1]: 1
Apr 18 17:23:51 hostname1 qdiskd[13247]: Assuming master role
  • When the qdisk master leaves in a master_wins configuration, quorum is lost on the remaining node and services are disabled rather than relocating to it.
  • Cannot fail over service when shutting down a node

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
  • cluster (cman, clusterlib) release prior to 3.0.12.1-49.el6
  • Use of a quorum disk (qdisk)
  • One of the following:
    • master_wins="1" set in the quorumd tag in /etc/cluster/cluster.conf
    • A 2-node cluster with no heuristics defined

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content