How to configure a stonith device using agent fence_vmware_rest in a RHEL 7, 8 or 9 High Availability cluster with pacemaker?
Environment
- Red Hat Enterprise Linux (RHEL) 7 Update 5 (with the High Availability Add On)
- Red Hat Enterprise Linux (RHEL) 8 or 9 (with the High Availability Add On)
Issue
How do I configure a stonith device using agent fence_vmware_rest
in a RHEL 7, 8 or 9 High Availability cluster with pacemaker?
Resolution
Assuming following is cluster architecture
- cluster node hostnames are
node1
andnode2
- cluster node names as seen by the vmware hypervisor (vCenter) are
node1-vm
andnode2-vm
List the Virtual Machine Names
-
Check if cluster node is able to reach the VMware vCenter and list the VMs that is managed by VMware vCenter. The following command will try to connect to VMware vCenter with the provided credentials and list all the VMs that is managed by VMware VCenter.
# fence_vmware_rest -a <vCenter IP address> -l <vcenter_username> -p <vcenter_password> --ssl-insecure -z -o list | egrep "(node1-vm|node2-vm)" node1-vm, node2-vm, # fence_vmware_rest -a <vCenter IP address> -l <vcenter_username> -p <vcenter_password> --ssl-insecure -z -o status -n node1-vm Status: ON
Verify that the status can be successful found for each VM that is a cluster node and test that the fence_vmware_rest
commands work on each VM that is a cluster node. If above command does not list any VMs or the command errors out then make sure the below is true.
- Verify the host (one of the cluster nodes) that is running the
fence_vmware_rest
command is able to communicate with VMware vCenter on port443/tcp
(when using SSL) or on port80/tcp
(without SSL). - Ensure that the user has permissions on VMware vCenter for fencing.
- Check if the VMware vCenter has trustworthy SSL certificate. If the certificate cannot be trustworthy checked then see the following solution on how to relax some SSL checks.
Add the stonith device to pacemaker
If the fence_vmware_rest
command succeeded in listing all the VMs managed by VMware vCenter and getting the status of the VMs that are cluster nodes then proceed on adding the the VMs managed by VMware vCenter topacemaker
.
The fence-agent fence_vmware_rest
is a shared fence-agent that uses the pcmk_host_map
attribute to correctly fence the cluster node. The pcmk_host_map
attribute is used to map node hostname as see by pacemaker
to the name of virtual machine as seen by VMware vCenter.
The first attribute in pcmk_host_map
is the cluster node name as seen in /etc/corosync/corosync.conf
file and the next attribute, that is post semicolon is the cluster node names as seen by the vmware hypervisor.
# cat /etc/corosync/corosync.conf
[...]
nodelist {
node {
ring0_addr: node1 <=== Cluster node name
nodeid: 1
}
node {
ring0_addr: node2 <=== Cluster node name
nodeid: 2
}
}
Add the fence_vmware_rest
stonith to pacemaker
:
# pcs stonith create vmfence fence_vmware_rest pcmk_host_map="node1:node1-vm;node2:node2-vm" ipaddr=<vCenter IP address> ssl=1 login=<vcenter_username> passwd=<vcenter_password> ssl_insecure=1
Check the status of stonith device
NOTE Prior to RHEL 8, replace pcs stonith status
with pcs stonith show
and replace pcs stonith config vmfence
with pcs stonith show vmfence --full
-
To check the status of stonith device and its configuration use the commands below.
# pcs stonith status * vmfence (stonith:fence_vmware_rest): Started node1
-
To show the configuration attributes of the stonith device.
# pcs stonith config vmfence Resource: vmfence (class=stonith type=fence_vmware_rest) Attributes: pcmk_host_map=node1:node1-vm;node2:node2-vm ipaddr=<vCenter IP address> ssl=1 login=<vcenter_username> passwd=<vcenter_password> ssl_insecure=1
Test that cluster nodes are fenced
After adding the stonith device then the stonith device should be tested that a cluster node can be successfully fenced. For more information on this then see the following article:
How to test fence devices and fencing configuration in a Red Hat High Availability cluster?
Additional notes and recommendations
- Make sure package
fence-agents-4.0.11-86.el7
or later is installed which has the fence-agent:fence_vmware_rest
. - Support Policies for RHEL High Availability Clusters - VMware Virtual Machines as Cluster Members
- Support Policies for RHEL High Availability clusters -
fence_vmware_rest
- Once configured, it is highly recommended to test that the cluster nodes can be successfully fence: How to test fence devices and fencing configuration in a Red Hat High Availability cluster?
- There is a known limitation imposed by the VMware Rest API of 1000 VMs:
fence_vmware_rest
monitor fails with error: "Exception: 400: Too many virtual machines. Add more filter criteria to reduce the number." - The fencing agent
fence_vmware_rest
currently does not support using UUIDs for VMs (only support using VM names). If you need to use UUID then usefence_vmware_soap
. - The fencing agent
fence_vmware_rest
works only with VMware vCenter because ESXi host does not provide the REST API. If you need to use ESXi host instead of VMware vCenter then usefence_vmware_soap
.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments