Chapter 6. Configuring fencing for an HA cluster on Red Hat OpenStack Platform

Fencing configuration ensures that a malfunctioning node on your HA cluster is automatically isolated. This prevents the node from consuming the cluster’s resources or compromising the cluster’s functionality.

Use the fence_openstack fence agent to configure a fence device for an HA cluster on RHOSP. You can view the options for the RHOSP fence agent with the following command.

# pcs stonith describe fence_openstack

Prerequisites

  • A configured HA cluster running on RHOSP
  • Access to the RHOSP APIs, using the RHOSP authentication method you will use for cluster configuration, as described in Setting up an authentication method for RHOSP
  • The cluster property stonith-enabled set to true, which is the default value. Red Hat does not support clusters when fencing is disabled, as it is not suitable for a production environment. Run the following command to ensure that fencing is enbaled.

    # pcs property config --all
    Cluster Properties:
    . . .
    stonith-enabled: true

Procedure

Complete the following steps from any node in the cluster.

  1. Determine the UUID for each node in your cluster.

    The following command displays the full list of all of the RHOSP instance names within the ha-example project along with the UUID for the cluster node associated with that RHOSP instance, under the heading ID. The node host name might not match the RHOSP instance name.

    # openstack --os-cloud="ha-example" server list
    …
    | ID                                  | Name             |...
    | 6d86fa7d-b31f-4f8a-895e-b3558df9decb|testnode-node03-vm|...
    | 43ed5fe8-6cc7-4af0-8acd-a4fea293bc62|testnode-node02-vm|...
    | 4df08e9d-2fa6-4c04-9e66-36a6f002250e|testnode-node01-vm|...
  2. Create the fencing device, using the pcmk_host_map parameter to map each node in the cluster to the UUID for that node. Each of the following example fence device creation commands uses a different authentication method.

    1. The following command creates a fence_openstack fencing device for a 3-node cluster, using a clouds.yaml configuration file for authentication. For the cloud= parameter, specify the name of the cloud in your clouds.yaml` file.

      # pcs stonith create fenceopenstack fence_openstack pcmk_host_map="node01:4df08e9d-2fa6-4c04-9e66-36a6f002250e;node02:43ed5fe8-6cc7-4af0-8acd-a4fea293bc62;node03:6d86fa7d-b31f-4f8a-895e-b3558df9decb" power_timeout="240" pcmk_reboot_timeout="480" pcmk_reboot_retries="4" cloud="ha-example"
    2. The following command creates a fence_openstack fencing device, using an OpenRC environment script for authentication.

      # pcs stonith create fenceopenstack fence_openstack pcmk_host_map="node01:4df08e9d-2fa6-4c04-9e66-36a6f002250e;node02:43ed5fe8-6cc7-4af0-8acd-a4fea293bc62;node03:6d86fa7d-b31f-4f8a-895e-b3558df9decb" power_timeout="240" pcmk_reboot_timeout="480" pcmk_reboot_retries="4" openrc="/root/openrc"
    3. The following command creates a fence_openstack fencing device, using a user name and password for authentication. The authentication parameters, including username, password, project_name, and auth_url, are provided by the RHOSP administrator.

      # pcs stonith create fenceopenstack fence_openstack pcmk_host_map="node01:4df08e9d-2fa6-4c04-9e66-36a6f002250e;node02:43ed5fe8-6cc7-4af0-8acd-a4fea293bc62;node03:6d86fa7d-b31f-4f8a-895e-b3558df9decb" power_timeout="240" pcmk_reboot_timeout="480" pcmk_reboot_retries="4" username="XXX" password="XXX" project_name="rhelha" auth_url="XXX" user_domain_name="Default"

Verification

  1. From one node in the cluster, fence a different node in the cluster and check the cluster status. If the fenced node is offline, the fencing operation was successful.

    [root@node01 ~] # pcs stonith fence node02
    [root@node01 ~] # pcs status
  2. Restart the node that you fenced and check the status to verify that the node started.

    [root@node01 ~] # pcs cluster start node02
    [root@node01 ~] # pcs status