10.5. Quorum Devices

Red Hat Enterprise Linux 7.4 provides full support for the ability to configure a separate quorum device which acts as a third-party arbitration device for the cluster. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow. A quorum device is recommended for clusters with an even number of nodes and highly recommended for two-node clusters.
You must take the following into account when configuring a quorum device.
  • It is recommended that a quorum device be run on a different physical network at the same site as the cluster that uses the quorum device. Ideally, the quorum device host should be in a separate rack than the main cluster, or at least on a separate PSU and not on the same network segment as the corosync ring or rings.
  • You cannot use more than one quorum device in a cluster at the same time.
  • Although you cannot use more than one quorum device in a cluster at the same time, a single quorum device may be used by several clusters at the same time. Each cluster using that quorum device can use different algorithms and quorum options, as those are stored on the cluster nodes themselves. For example, a single quorum device can be used by one cluster with an ffsplit (fifty/fifty split) algorithm and by a second cluster with an lms (last man standing) algorithm.
  • A quorum device should not be run on an existing cluster node.

10.5.1. Installing Quorum Device Packages

Configuring a quorum device for a cluster requires that you install the following packages:
  • Install corosync-qdevice on the cluster nodes.
    [root@node1:~]# yum install corosync-qdevice
    [root@node2:~]# yum install corosync-qdevice
  • Install corosync-qnetd on the quorum device host.
    [root@qdevice:~]# yum install corosync-qnetd

10.5.2. Configuring a Quorum Device

This section provides a sample procedure to configure a quorum device in a Red Hat high availability cluster. The following procedure configures a quorum device and adds it to the cluster. In this example:
  • The node used for a quorum device is qdevice.
  • The quorum device model is net, which is currently the only supported model. The net model supports the following algorithms:
    • ffsplit: fifty-fifty split. This provides exactly one vote to the partition with the highest number of active nodes.
    • lms: last-man-standing. If the node is the only one left in the cluster that can see the qnetd server, then it returns a vote.

      Warning

      The LMS algorithm allows the cluster to remain quorate even with only one remaining node, but it also means that the voting power of the quorum device is great since it is the same as number_of_nodes - 1. Losing connection with the quorum device means losing number_of_nodes - 1 votes, which means that only a cluster with all nodes active can remain quorate (by overvoting the quorum device); any other cluster becomes inquorate.
    For more detailed information on the implementation of these algorithms, see the corosync-qdevice(8) man page.
  • The cluster nodes are node1 and node2.
The following procedure configures a quorum device and adds that quorum device to a cluster.
  1. On the node that you will use to host your quorum device, configure the quorum device with the following command. This command configures and starts the quorum device model net and configures the device to start on boot.
    [root@qdevice:~]# pcs qdevice setup model net --enable --start
    Quorum device 'net' initialized
    quorum device enabled
    Starting quorum device...
    quorum device started
    
    After configuring the quorum device, you can check its status. This should show that the corosync-qnetd daemon is running and, at this point, there are no clients connected to it. The --full command option provides detailed output.
    [root@qdevice:~]# pcs qdevice status net --full
    QNetd address:                  *:5403
    TLS:                            Supported (client certificate required)
    Connected clients:              0
    Connected clusters:             0
    Maximum send/receive size:      32768/32768 bytes
    
  2. From one of the nodes in the existing cluster, authenticate user hacluster on the node that is hosting the quorum device.
    [root@node1:~] # pcs cluster auth qdevice
    Username: hacluster
    Password:
    qdevice: Authorized
    
  3. Add the quorum device to the cluster.
    Before adding the quorum device, you can check the current configuration and status for the quorum device for later comparison. The output for these commands indicates that the cluster is not yet using a quorum device.
    [root@node1:~]# pcs quorum config
    Options:
    
    [root@node1:~]# pcs quorum status
    Quorum information
    ------------------
    Date:             Wed Jun 29 13:15:36 2016
    Quorum provider:  corosync_votequorum
    Nodes:            2
    Node ID:          1
    Ring ID:          1/8272
    Quorate:          Yes
    
    Votequorum information
    ----------------------
    Expected votes:   2
    Highest expected: 2
    Total votes:      2
    Quorum:           1  
    Flags:            2Node Quorate 
    
    Membership information
    ----------------------
        Nodeid      Votes    Qdevice Name
             1          1         NR node1 (local)
             2          1         NR node2
    
    
    The following command adds the quorum device that you have previously created to the cluster. You cannot use more than one quorum device in a cluster at the same time. However, one quorum device can be used by several clusters at the same time. This example command configures the quorum device to use the ffsplit algorithm. For information on the configuration options for the quorum device, see the corosync-qdevice(8) man page.
    [root@node1:~]# pcs quorum device add model net host=qdevice algorithm=ffsplit
    Setting up qdevice certificates on nodes...
    node2: Succeeded
    node1: Succeeded
    Enabling corosync-qdevice...
    node1: corosync-qdevice enabled
    node2: corosync-qdevice enabled
    Sending updated corosync.conf to nodes...
    node1: Succeeded
    node2: Succeeded
    Corosync configuration reloaded
    Starting corosync-qdevice...
    node1: corosync-qdevice started
    node2: corosync-qdevice started
    
  4. Check the configuration status of the quorum device.
    From the cluster side, you can execute the following commands to see how the configuration has changed.
    The pcs quorum config shows the quorum device that has been configured.
    [root@node1:~]# pcs quorum config
    Options:
    Device:
      Model: net
        algorithm: ffsplit
        host: qdevice
    
    The pcs quorum status command shows the quorum runtime status, indicating that the quorum device is in use.
    [root@node1:~]# pcs quorum status
    Quorum information
    ------------------
    Date:             Wed Jun 29 13:17:02 2016
    Quorum provider:  corosync_votequorum
    Nodes:            2
    Node ID:          1
    Ring ID:          1/8272
    Quorate:          Yes
    
    Votequorum information
    ----------------------
    Expected votes:   3
    Highest expected: 3
    Total votes:      3
    Quorum:           2  
    Flags:            Quorate Qdevice 
    
    Membership information
    ----------------------
        Nodeid      Votes    Qdevice Name
             1          1    A,V,NMW node1 (local)
             2          1    A,V,NMW node2
             0          1            Qdevice
    
    The pcs quorum device status shows the quorum device runtime status.
    [root@node1:~]# pcs quorum device status
    Qdevice information
    -------------------
    Model:                  Net
    Node ID:                1
    Configured node list:
        0   Node ID = 1
        1   Node ID = 2
    Membership node list:   1, 2
    
    Qdevice-net information
    ----------------------
    Cluster name:           mycluster
    QNetd host:             qdevice:5403
    Algorithm:              ffsplit
    Tie-breaker:            Node with lowest node ID
    State:                  Connected
    
    From the quorum device side, you can execute the following status command, which shows the status of the corosync-qnetd daemon.
    [root@qdevice:~]# pcs qdevice status net --full
    QNetd address:                  *:5403
    TLS:                            Supported (client certificate required)
    Connected clients:              2
    Connected clusters:             1
    Maximum send/receive size:      32768/32768 bytes
    Cluster "mycluster":
        Algorithm:          ffsplit
        Tie-breaker:        Node with lowest node ID
        Node ID 2:
            Client address:         ::ffff:192.168.122.122:50028
            HB interval:            8000ms
            Configured node list:   1, 2
            Ring ID:                1.2050
            Membership node list:   1, 2
            TLS active:             Yes (client certificate verified)
            Vote:                   ACK (ACK)
        Node ID 1:
            Client address:         ::ffff:192.168.122.121:48786
            HB interval:            8000ms
            Configured node list:   1, 2
            Ring ID:                1.2050
            Membership node list:   1, 2
            TLS active:             Yes (client certificate verified)
            Vote:                   ACK (ACK)
    

10.5.3. Managing the Quorum Device Service

PCS provides the ability to manage the quorum device service on the local host (corosync-qnetd), as shown in the following example commands. Note that these commands affect only the corosync-qnetd service.
[root@qdevice:~]# pcs qdevice start net
[root@qdevice:~]# pcs qdevice stop net
[root@qdevice:~]# pcs qdevice enable net
[root@qdevice:~]# pcs qdevice disable net
[root@qdevice:~]# pcs qdevice kill net

10.5.4. Managing the Quorum Device Settings in a Cluster

The following sections describe the PCS commands that you can use to manage the quorum device settings in a cluster, showing examples that are based on the quorum device configuration in Section 10.5.2, “Configuring a Quorum Device”.

10.5.4.1. Changing Quorum Device Settings

You can change the setting of a quorum device with the pcs quorum device update command.

Warning

To change the host option of quorum device model net, use the pcs quorum device remove and the pcs quorum device add commands to set up the configuration properly, unless the old and the new host are the same machine.
The following command changes the quorum device algorithm to lms.
[root@node1:~]# pcs quorum device update model algorithm=lms
Sending updated corosync.conf to nodes...
node1: Succeeded
node2: Succeeded
Corosync configuration reloaded
Reloading qdevice configuration on nodes...
node1: corosync-qdevice stopped
node2: corosync-qdevice stopped
node1: corosync-qdevice started
node2: corosync-qdevice started

10.5.4.2. Removing a Quorum Device

Use the following command to remove a quorum device configured on a cluster node.
[root@node1:~]# pcs quorum device remove
Sending updated corosync.conf to nodes...
node1: Succeeded
node2: Succeeded
Corosync configuration reloaded
Disabling corosync-qdevice...
node1: corosync-qdevice disabled
node2: corosync-qdevice disabled
Stopping corosync-qdevice...
node1: corosync-qdevice stopped
node2: corosync-qdevice stopped
Removing qdevice certificates from nodes...
node1: Succeeded
node2: Succeeded
After you have removed a quorum device, you should see the following error message when displaying the quorum device status.
[root@node1:~]# pcs quorum device status
Error: Unable to get quorum status: corosync-qdevice-tool: Can't connect to QDevice socket (is QDevice running?): No such file or directory

10.5.4.3. Destroying a Quorum Device

To disable and stop a quorum device on the quorum device host and delete all of its configuration files, use the following command.
[root@qdevice:~]# pcs qdevice destroy net
Stopping quorum device...
quorum device stopped
quorum device disabled
Quorum device 'net' configuration files removed