High Availability for Compute Instances
Configure High Availability for Compute Instances
Chapter 1. Overview
This guide describes how to manage Instance High Availability (Instance HA). Instance HA allows Red Hat OpenStack Platform to automatically evacuate and re-spawn instances on a different Compute node when their host Compute node fails.
The evacuation process that is triggered by Instance HA is similar to what users can do manually, as described in Evacuate Instances.
Instance HA works on shared storage or local storage environments, which means that evacuated instances maintain the same network configuration (static IP, floating IP, and so on) and the same characteristics inside the new host, even if they are spawned from scratch.
Instance HA is managed by the following resource agents:
|Agent name||Name inside cluster||Role|
| || || |
Marks a Compute node for evacuation when the node becomes unavailable.
| || || |
Evacuates instances from failed nodes. This agent runs on one of the Controller nodes.
| || || |
Releases a fenced node and enables the node to run instances again.
Chapter 2. How Instance HA Works
OpenStack uses Instance HA to automate the process of evacuating instances from a Compute node when that node fails. The following procedure describes the sequence of events that are triggered when a Compute node fails.
At the time of failure, the
IPMIagent performs first-layer fencing and physically resets the node to ensure that it is powered off. Evacuating instances from online Compute nodes might result in data corruption or in multiple identical instances running on the overcloud. When the node is powered off, it is considered fenced.
After the physical IPMI fencing, the
fence-novaagent performs second-layer fencing and marks the fenced node with the
“evacuate=yes”cluster per-node attribute. To do this, the agent runs the following command:
$ attrd_updater -n evacuate -A name="evacuate" host="FAILEDHOST" value="yes"
Where FAILEDHOST is the hostname of the failed Compute node.
nova-evacuateagent continually runs in the background, periodically checking the cluster for nodes with the
nova-evacuatedetects that the fenced node contains this attribute, the agent starts evacuating the node using the process described in Evacuate Instances.
While the failed node boots up from the IPMI reset, the
nova-computeprocess on that node starts automatically. Because the node was fenced earlier, it does not run any new instances until Pacemaker unfences it.
When Pacemaker sees that the Compute node is online again, it tries to start the
compute-unfence-triggerresource on the node, reverting the force-down API call and setting the node as enabled again.
2.1. Designating specific instances to be evacuated
By default, all instances are to be evacuated, but it is also possible to tag images or flavors for evacuation.
To tag an image:
$ openstack image set --tag evacuable ID-OF-THE-IMAGE
To tag a flavor:
$ nova flavor-key ID-OF-THE-FLAVOR set evacuable=true
Chapter 3. Installing and configuring Instance HA
Instance HA is deployed and configured with the director. However, there are a few additional steps that you need to perform to prepare for the deployment.
This section includes all the steps needed to configure a new Instance HA deployment on a new overcloud with the goal of enabling Instance HA on a subset of Compute nodes with a custom role.
- Compute node host names and Pacemaker remote resource names must comply with the W3C naming conventions. For more information, see Declaring Namespaces and Names and Tokens in the W3C documentation.
If you deploy Instance HA in a Spine-Leaf environment, you must define a single
internal_apinetwork for the Controller and Compute nodes. You can then define a subnet for each leaf. For more information about configuring Spine-Leaf networks, see Creating a roles data file in the Spine Leaf Networking guide.
- If you want to enable instance HA in a different environment, such as an existing overcloud using either standard or custom roles, follow only the procedures that are relevant to your deployment and adapt your templates accordingly.
- Disabling Instance HA with the director after installation is not supported. For a workaround to manually remove Instance HA components from your deployment, see the article How can I remove Instance HA components from the controller nodes? . This workaround is provided as-is. You must verify the procedure in a test environment before implementing in production.
Configure the Instance HA role, flavor, and profile
ComputeInstanceHArole to your roles data file and regenerate the file. For example:
$ openstack overcloud roles generate -o ~/my_roles_data.yaml Controller Compute ComputeInstanceHA
ComputeInstanceHArole includes all the services in the default
Computerole as well as the
PacemakerRemoteservices. For general information about custom roles and about the roles-data.yaml, see the Roles section in the Advanced Overcloud Customization guide.
compute-instance-haflavor to tag Compute nodes that you want to designate for Instance HA. For example:
$ source ~/stackrc $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-instance-ha $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute-instance-ha" compute-instance-ha
Tag each Compute node that you want to designate for Instance HA with the
$ openstack baremetal node set --property capabilities='profile:compute-instance-ha,boot_option:local' <NODE UUID>
ComputeInstanceHArole to the
compute-instance-haflavor by creating an environment file with the following content:
parameter_defaults: OvercloudComputeInstanceHAFlavor: compute-instance-ha
Enable fencing on all Controller and Compute nodes in the overcloud by creating an environment file with fencing information. Make sure to create the environment file in an accessible location, such as ~/templates. For example:
parameter_defaults: EnableFencing: true FencingConfig: devices: - agent: fence_ipmilan host_mac: 00:ec:ad:cb:3c:c7 params: login: admin ipaddr: 192.168.24.1 ipport: 6230 passwd: password lanplus: 1 - agent: fence_ipmilan host_mac: 00:ec:ad:cb:3c:cb params: login: admin ipaddr: 192.168.24.1 ipport: 6231 passwd: password lanplus: 1 - agent: fence_ipmilan host_mac: 00:ec:ad:cb:3c:cf params: login: admin ipaddr: 192.168.24.1 ipport: 6232 passwd: password lanplus: 1 - agent: fence_ipmilan host_mac: 00:ec:ad:cb:3c:d3 params: login: admin ipaddr: 192.168.24.1 ipport: 6233 passwd: password lanplus: 1 - agent: fence_ipmilan host_mac: 00:ec:ad:cb:3c:d7 params: login: admin ipaddr: 192.168.24.1 ipport: 6234 passwd: password lanplus: 1
For more information about fencing and STONITH configuration, see the Fencing Controller Nodes with STONITH section of the High Availability Deployment and Usage guide.
Instance HA uses shared storage by default. If shared storage is not configured for your Compute instance, then add the following parameter to the environment file that you created in the previous step:
parameter_defaults: ExtraConfig: tripleo::instanceha::no_shared_storage: true
See Section 3.1, “Considerations for Shared Storage” for details on how to boot from an OpenStack Block Storage (cinder) volume rather than configuring shared storage for the disk image of instances.
Deploy the overcloud
openstack overcloud deploy command with the
-e option for each environment file that you created, as well as the compute-instanceha.yaml environment file. For example:
$ openstack overcloud deploy --templates \ -e <FLAVOR_ENV_FILE> \ -e <FENCING_ENV_FILE> \ -e /usr/share/openstack-tripleo-heat-templates/environments/compute-instanceha.yaml
- Do not modify the compute-instanceha.yaml environment file.
- Make sure to include the path to each environment file that you want to include in the overcloud deployment.
You can configure Instance HA for your overcloud at any time after creating the undercloud. If you already deployed the overcloud, you need to rerun the
overcloud deploycommand with the new Instance HA files.
After the deployment is complete, each Compute node will include a
STONITH device and a
3.1. Considerations for Shared Storage
Typically, Instance HA requires that you configure shared storage for disk images of instances. Therefore, if you attempt to use the
no-shared-storage option, you might receive an
InvalidSharedStorage error during evacuation, and the instances will not boot on another Compute node.
However, if all your instances are configured to boot from an OpenStack Block Storage (
cinder) volume, you do not need to configure shared storage for the disk image of instances, and you can still evacuate all instances using the
During evacuation, if your instances are configured to boot from a Block Storage volume, any evacuated instances should boot from the same volume on another Compute node. As a result, the evacuated instances immediately restart their jobs because the OS image and the application data are stored on the OpenStack Block Storage volume..
Chapter 4. Testing Evacuation with Instance HA
The following procedure involves deliberately crashing a Compute node. Doing this forces the automated evacuation of instances through Instance HA.
Boot one or more instances on the overcloud before crashing the Compute node that hosts the instances to test.
stack@director $ . overcloudrc stack@director $ nova boot --image cirros --flavor 2 test-failover stack@director $ nova list --fields name,status,host
Log in to the Compute node that hosts the instances, using the
stack@director $ . stackrc stack@director $ ssh -l heat-admin compute-n heat-admin@compute-n $
Crash the Compute node.
heat-admin@compute-n $ echo c > /proc/sysrq-trigger
Wait a few minutes and then verify that these instances re-spawned on another Compute nodes.
stack@director $ nova list --fields name,status,host stack@director $ nova service-list