How to configure a stonith device using agent fence_vmware_rest in a RHEL 7, 8 or 9 High Availability cluster with pacemaker?

Resolution

Assuming following is cluster architecture

• cluster node hostnames are node1 and node2
• cluster node names as seen by the vmware hypervisor (vCenter) are node1-vm and node2-vm

List the Virtual Machine Names

• Check if cluster node is able to reach the VMware vCenter and list the VMs that is managed by VMware vCenter. The following command will try to connect to VMware vCenter with the provided credentials and list all the VMs that is managed by VMware VCenter.

   # fence_vmware_rest -a <vCenter IP address> -l <vcenter_username> -p <vcenter_password> --ssl-insecure -z -o list | egrep "(node1-vm|node2-vm)"
   node1-vm,
   node2-vm,
   # fence_vmware_rest -a <vCenter IP address> -l <vcenter_username> -p <vcenter_password> --ssl-insecure -z -o status -n node1-vm
   Status: ON
  

Verify that the status can be successful found for each VM that is a cluster node and test that the fence_vmware_rest commands work on each VM that is a cluster node. If above command does not list any VMs or the command errors out then make sure the below is true.

• Verify the host (one of the cluster nodes) that is running the fence_vmware_rest command is able to communicate with VMware vCenter on port 443/tcp (when using SSL) or on port 80/tcp (without SSL).
• Ensure that the user has permissions on VMware vCenter for fencing.
• Check if the VMware vCenter has trustworthy SSL certificate. If the certificate cannot be trustworthy checked then see the following solution on how to relax some SSL checks.

Add the stonith device to pacemaker

If the fence_vmware_rest command succeeded in listing all the VMs managed by VMware vCenter and getting the status of the VMs that are cluster nodes then proceed on adding the the VMs managed by VMware vCenter topacemaker.

The fence-agent fence_vmware_rest is a shared fence-agent that uses the pcmk_host_map attribute to correctly fence the cluster node. The pcmk_host_map attribute is used to map node hostname as see by pacemaker to the name of virtual machine as seen by VMware vCenter.

The first attribute in pcmk_host_map is the cluster node name as seen in /etc/corosync/corosync.conf file and the next attribute, that is post semicolon is the cluster node names as seen by the vmware hypervisor.

    # cat /etc/corosync/corosync.conf
    [...]
    nodelist {
        node {
            ring0_addr: node1  <=== Cluster node name
            nodeid: 1
        }

        node {
            ring0_addr: node2  <=== Cluster node name
            nodeid: 2
        }
    }

Add the fence_vmware_rest stonith to pacemaker:

# pcs stonith create vmfence fence_vmware_rest pcmk_host_map="node1:node1-vm;node2:node2-vm" ipaddr=<vCenter IP address> ssl=1 login=<vcenter_username> passwd=<vcenter_password> ssl_insecure=1

Check the status of stonith device

NOTE Prior to RHEL 8, replace pcs stonith status with pcs stonith show and replace pcs stonith config vmfence with pcs stonith show vmfence --full

• To check the status of stonith device and its configuration use the commands below.

  # pcs stonith status
  * vmfence   (stonith:fence_vmware_rest):    Started node1
  

• To show the configuration attributes of the stonith device.

  # pcs stonith config vmfence 
   Resource: vmfence (class=stonith type=fence_vmware_rest)
    Attributes: pcmk_host_map=node1:node1-vm;node2:node2-vm ipaddr=<vCenter IP address> ssl=1 login=<vcenter_username> passwd=<vcenter_password> ssl_insecure=1
  

Test that cluster nodes are fenced

After adding the stonith device then the stonith device should be tested that a cluster node can be successfully fenced. For more information on this then see the following article:
How to test fence devices and fencing configuration in a Red Hat High Availability cluster?

Additional notes and recommendations

• Make sure package fence-agents-4.0.11-86.el7 or later is installed which has the fence-agent: fence_vmware_rest.
• Support Policies for RHEL High Availability Clusters - VMware Virtual Machines as Cluster Members
• Support Policies for RHEL High Availability clusters - fence_vmware_rest
• Once configured, it is highly recommended to test that the cluster nodes can be successfully fence: How to test fence devices and fencing configuration in a Red Hat High Availability cluster?
• There is a known limitation imposed by the VMware Rest API of 1000 VMs: fence_vmware_rest monitor fails with error: "Exception: 400: Too many virtual machines. Add more filter criteria to reduce the number."
• The fencing agent fence_vmware_rest currently does not support using UUIDs for VMs (only support using VM names). If you need to use UUID then use fence_vmware_soap.
• The fencing agent fence_vmware_rest works only with VMware vCenter because ESXi host does not provide the REST API. If you need to use ESXi host instead of VMware vCenter then use fence_vmware_soap.