Does a quorum disk require a heuristic be defined on RHEL 5 and RHEL 6?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux Server 5 (with the High Availability Add Ons)
  • Red Hat Enterprise Linux Server 6 (with the High Availability Add Ons)

Issue

  • Does a quorum disk require a heuristic be defined on RHEL 5 and RHEL 6?

Resolution

A heuristic has to be defined for all quorum disk configurations except when using the option master_wins for 2 node clusters. An example heuristic that is commonly used is to ping the gateway that is used by the cluster nodes heartbeat interface, including the -w option to ensure the heuristic does not time out unnecessarily:

  <heuristic interval="2" program="ping -c1 -w1 192.168.2.1" score="1" tko="3"/>

NOTE: In situations where a network split can result in two quorate sides within the cluster, the heuristics must be configured such that one side of the split will drop below min_score. The scenarios in which such a split can occur are:

  • A two node cluster with a quorum device that uses heuristics rather than master_wins mode
  • A cluster of more than two nodes which uses a "last man standing" quorum device configuration.

The required heuristic must be designed such that it fails for one side of the split membership. This can be accomplished by configuring a network-checking heuristic (such as the example above) on the cluster-interconnect network, and configuring its score and min_score such that the failure of this heuristic will cause a drop below min_score. For example, in /etc/cluster/cluster.conf:

[...]
<clusternodes>
   <clusternode name="node1.example.com" votes="1" nodeid="1">
   [...]
   <clusternode name="node2.example.com" votes="1" nodeid="2">
   [...]
   <clusternode name="node3.example.com" votes="1" nodeid="3">
   [...]
</clusternodes>
<cman expected_votes="5"/>
<totem token="21000"/>
<quorumd label="myqdisk" votes="2" min_score="1">
   <heuristic score="1" program="ping -c1 -w1 192.168.2.1." interval="2" tko="4"/>
</quorumd>
[...]

This example quorum-device configuration is for a cluster which communicates over the network with 192.168.2.1 as a gateway. If node1 loses connectivity on that network, its heuristic will fail, causing a drop below min_score which will result in a reboot. This will automatically resolve the split situation between node1 and nodes2/3, avoiding any fence race or split-brain scenario.

Root Cause

A <quorumd> section in the /etc/cluster/cluster.conf that is not using the option master_wins for 2 node cluster are required to have a heuristic defined.

In situations where a network split can result in two independent sets of nodes maintaining quorum, there is the potential for a fence race that can take down the entire cluster. For this reason, any heuristics that are implemented must ensure that such a split will be avoided. This can be done by ensuring the total heuristic score drops below min_score in the event of a split.

The example provided above avoids network splits by detecting a network failure and causing the node(s) having issues to reboot. The scenarios in which this would be needed are:

  • A two-node cluster with a quorum device, not using master_wins. In this situation, if either node loses connectivity, then both sides will see themself as the only remaining node; but in combination with the votes from the quorum device, both sides would still have quorum, and thus would attempt to fence each other simultaneously. As such, the heuristic(s) must be designed such that this network failure results in the total score falling below min_score so that the node is rebooted, avoiding the split and fence race.

  • A cluster with more than two nodes and a quorum device configured for last-man-standing. In such a configuration, if one node loses connectivity it would still maintain quorum if the quorum device remained accessible; the other nodes would also maintain quorum because they too have access to the quorum device, and as such the membership decision would be left to cman, which would result in a split where the two sides could race to fence each other. As such, the heuristic must result in one side or the other "losing", dropping the heuristic score and causing a reboot so the other side can remain functional by itself until the issue is resolved.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments