Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 8. Migrating Virtual Machines Between Compute Nodes

In some situations, you might need to migrate virtual machines from one Compute node to another Compute node in the overcloud. For example:

  • Compute Node Maintenance: If you must temporarily take a Compute node out of service, you can temporarily migrate virtual machines running on the Compute node to another Compute node. Common scenarios include hardware maintenance, hardware repair, kernel upgrades and software updates.
  • Failing Compute Node: If a Compute node is about to fail and must be serviced or replaced, you must migrate virtual machines from the failing Compute node to a healthy Compute node. For Compute nodes that have already failed, see Evacuating VMs.
  • Workload Rebalancing: You can consider migrating one or more virtual machines to another Compute node to rebalance the workload. For example, you can consolidate virtual machines on a Compute node to conserve power, migrate virtual machines to a Compute node that is physically closer to other networked resources to reduce latency, or distribute virtual machines across Compute nodes to avoid hot spots and increase resiliency.

The director configures all Compute nodes to provide secure migration. All Compute nodes also require a shared SSH key to provide each host’s nova user with access to other Compute nodes during the migration process. The director creates this key using the OS::TripleO::Services::NovaCompute composable service. This composable service is one of the main services included on all Compute roles by default (see Composable Services and Custom Roles in Advanced Overcloud Customization).

Updates to Migration Functionality

The latest update of Red Hat OpenStack Platform 10 includes the OS::TripleO::Services::Sshd composable service, which is required for live migration capabilities. The director’s core template collection did not include this service in the initial release but is now included in the openstack-tripleo-heat-templates-5.2.0-12 package and later versions.

  • If using the default roles, update your environment to use the Heat templates from the openstack-tripleo-heat-templates-5.2.0-12 package or later versions.
  • If using a custom roles data file, update your environment to use the Heat templates from the openstack-tripleo-heat-templates-5.2.0-12 package or later versions, include the OS::TripleO::Services::Sshd service on each overcloud role, then update your overcloud stack to include the new service.

For more information, see "Red Hat OpenStack Platform director (TripleO) CVE-2017-2637 bug and Red Hat OpenStack Platform".

Important

RHSA-2018:0369 - Security Advisory issued on 2018-02-27 and RHBA-2019:0074 - Bug Fix Advisory issued on 2019-01-16 impact the migration of virtual machines.

RHSA-2018:0369 - Security Advisory prevents live migration of virtual machines to Compute nodes where the CPUs defined in the domain XML are already in use. RHSA-2018:0369 - Security Advisory unnecessarily imposes the same restrictions on operations such as cold migration, evacuation, resize and unshelve. RHBA-2019:0074 - Bug Fix Advisory relaxes those restrictions by making them optional. See the new cpu_pinning_migration_quick_fail configuration option introduced with this advisory for details.

Red Hat recommends applying both advisories.

See Migration Constraints for additional details on migration constraints.

8.1. Migration Types

OpenStack Platform supports two types of migration:

Live Migration

Live migration involves spinning up the virtual machine on the destination node and shutting down the virtual machine on the source node seamlessly while maintaining state consistency.

Live Migration

Live migration handles virtual machine migration with little or no perceptible downtime. In some cases, virtual machines cannot use live migration. See Migration Constraints for details on migration constraints.

Cold Migration

Cold migration or non-live migration involves nova shutting down a virtual machine before migrating it from the source Compute node to the destination Compute node.

Cold Migration

Cold migration involves some downtime for the virtual machine. However, cold migration still provides the migrated virtual machine with access to the same volumes and IP addresses.

Important

For source Compute nodes that have already failed, see Evacuation. Migration requires that both the source and destination Compute nodes are running.

8.2. Migration Constraints

In some cases, migrating virtual machines involves additional constraints. Migration constraints typically arise with block migration, configuration disks, or when one or more virtual machines access physical hardware on the Compute node.

CPU Constraints

The source and destination Compute nodes must have the same CPU architecture. For example, Red Hat does not support migrating a virtual machine from an x86_64 CPU to a ppc64le CPU. In some cases, the CPU of the source and destination Compute node must match exactly, such as virtual machines using CPU host passthrough. In all cases, the CPU features of the destination node must be a superset of the CPU features on the source node. Additionally, when virtual machines use CPU pinning, the NUMA node used on the source node must target the same NUMA node on the destination Compute node and the NUMA node must be empty.

Memory Constraints

The destination Compute node must have sufficient available RAM. Memory oversubscription can cause migration to fail. Additionally, virtual machines using a NUMA topology must have sufficient available RAM on the same NUMA node on the destination Compute node.

Block Migration Constraints

Migrating virtual machines that use disks stored locally on a Compute node takes significantly longer than migrating volume-backed virtual machines that utilize shared storage such as Red Hat Ceph Storage. This latency arises because nova migrates local disks block-by-block between the Compute nodes over the control plane network by default. By contrast, volume-backed instances using shared storage such as Red Hat Ceph Storage do not have to migrate the volumes, because each Compute node already has access to the shared storage.

Note

Network congestion in the control plane network caused by migrating local disks or virtual machines that consume large amounts of RAM could impact the performance of other systems using the control plane network, such as RabbitMQ.

Read-only Drive Migration Constraints

Migrating a drive is only supported if the drive has both read and write capabilities. For example, nova cannot migrate a CD-ROM drive or a read-only config drive. However, nova can migrate a drive with both read and write capabilities, including a config drive with a drive format such as vfat.

Live Migration Constraints

There are a few additional live migration constraints in Red Hat OpenStack Platform:

  • No New Operations During Migration: To achieve state consistency between the copies of the virtual machine on the source and destination nodes, Red Hat OpenStack Platform must prevent new operations during live migration. Otherwise, live migration could take a long time or potentially never end if writes to memory occur faster than live migration can replicate the state of the memory.
  • Non-Uniform Memory Access (NUMA): In previous releases, migrating virtual machines when configured with a NUMA topology was not recommended. Currently, nova can migrate virtual machines with NUMA topology cleanly subject to some constraints.
  • CPU Pinning: When a flavor uses CPU pinning, the flavor implicitly introduces a NUMA topology to the virtual machine and maps its CPUs and memory to specific host CPUs and memory. The difference been a simple NUMA topology and CPU pinning is that NUMA uses a range of CPU cores, whereas CPU pinning uses specific CPU cores. See CPU pinning for additional details.
  • Data Plane Development Kit (DPDK): When a virtual machine uses DPDK, such as a virtual machine running Open vSwitch with dpdk-netdev, the virtual machine also uses huge pages which imposes a NUMA topology such that nova pins the virtual machine to a NUMA node.

nova can live migrate a virtual machine that uses NUMA, CPU pinning or DPDK. However, the destination Compute node must have sufficient capacity on the same NUMA node that the virtual machine uses on the source Compute node. For example, if a virtual machine uses NUMA 0 on overcloud-compute-0, when migrating the virtual machine to overcloud-compute-1, you must ensure that overcloud-compute-1 has sufficient capacity on NUMA 0 to support the virtual machine in order to use live migration.

Constraints that Preclude Live Migration

There are a few cases where virtual machine configuration precludes live migration in Red Hat OpenStack Platform:

  • Single-root Input/Output Virtualization (SR-IOV): You can assign SR-IOV Virtual Functions (VFs) to virtual machines. However, this prevents live migration. Unlike a regular network device, an SR-IOV VF network device does not have a permanent unique MAC address. The VF network device receives a new MAC address each time the Compute node reboots or when nova-scheduler migrates the virtual machine to a new Compute node. Consequently, nova cannot live migrate virtual machines that use SR-IOV in OpenStack Platform 10. You must cold migrate virtual machines that use SR-IOV.
  • PCI Passthrough: QEMU/KVM hypervisors support attaching PCI devices on the Compute node to a virtual machine. PCI passthrough allows a virtual machine to have exclusive access to PCI devices, which appear and behave as if they are physically attached to the virtual machine’s operating system. However, since PCI passthrough involves physical addresses, nova does not support live migration of virtual machines using PCI passthrough in OpenStack Platform 10.

8.3. Pre-migration Procedures

Before migrating one or more virtual machines, perform the following steps:

Procedure

  1. From the undercloud, identify the source Compute node hostname and the destination Compute node hostname. 

    $ source ~/overcloudrc
$ openstack compute service list

  2. List virtual machines on the source Compute node and locate the ID of the virtual machine(s) you intend to migrate: 

    $ openstack server list --host [source] --all-projects

    Replace [source] with the host name of the source Compute node.

Pre-Migration Procedure for Compute Node Maintenance

If you are taking down the source Compute node for maintenance, disable the source Compute node from the undercloud to ensure that the scheduler does not attempt to assign new virtual machines to the source Compute node during maintenance. 

$ openstack compute service set [source] nova-compute --disable

Replace [source] with the host name of the source Compute node.

Pre-Migration Procedure for NUMA, CPU-pinned and DPDK Instances

When migrating virtual machines that use NUMA, CPU-pinning or DPDK, the destination Compute node should have an identical hardware specification and configuration as the source Compute node. Additionally, the destination Compute node should have no virtual machines running on it to ensure that it preserves the NUMA topology of the source Compute node.

Note

When migrating virtual machines using NUMA, CPU-pinning or DPDK, the /etc/nova/nova.conf file requires appropriate values for the scheduler_default_filters configuration setting, such as AggregateInstanceExtraSpecFilter and NUMATopologyFilter. You can accomplish this by setting the NovaSchedulerDefaultFilters heat parameter in an environment file.

  1. If the destination Compute node for NUMA, CPU-pinned or DPDK virtual machines is not disabled, disable it to prevent the scheduler from assigning virtual machines to the node. 

    $ openstack compute service set [dest] nova-compute --disable

    Replace [dest] with the host name of the destination Compute node.

  2. Ensure the destination Compute node has no virtual machines, except for virtual machines previously migrated from the source Compute node when migrating multiple DPDK or NUMA virtual machines. 

    $ openstack server list --host [dest] --all-projects

    Replace [dest] with the host name of the destination Compute node.

  3. Ensure the destination Compute node has sufficient resources to run the NUMA, CPU-pinned or DPDK virtual machine. 

    $ openstack host show overcloud-compute-n
$ ssh overcloud-compute-n
$ numactl --hardware
$ exit

    Replace overcloud-compute-n with the host name of the destination Compute node.

  4. To discover NUMA information about the source or destination Compute nodes, execute the following as needed: 

    $ ssh root@overcloud-compute-n
# lscpu && lscpu | grep NUMA
# virsh nodeinfo
# virsh capabilities
# exit

    Use ssh to connect to overcloud-compute-n where overcloud-compute-n is the source or destination Compute node.

  5. If you are unsure if a virtual machine uses NUMA, check the flavor of the virtual machine. 

    $ openstack server list -c Name -c Flavor --name [vm]

    Replace [vm] with the name or ID for the virtual machine.

    Then, check the flavor: 

    $ openstack flavor show [flavor]

    Replace [flavor] with the name or ID of the flavor. If the result of the properties field includes hw:mem_page_size with a value other than any such as 2MB, 2048 or 1GB, the virtual machine will have a NUMA topology. If the properties field includes aggregate_instance_extra_specs:pinned='true', the virtual machine uses CPU pinning. If the properties field includes hw:numa_nodes, nova restricts the virtual machine to a specific NUMA node.

  6. For each virtual machine using NUMA, consider retrieving information about the NUMA topology from the underlying Compute node so that you can verify that the NUMA topology on the destination Compute node reflects the NUMA topology of the source Compute node after migration is complete. 

    $ ssh root@overcloud-compute-n
# virsh vcpuinfo [vm]
# virsh numatune [vm]
# exit

    Replace [vm] with the name of the virtual machine. The vcpuinfo command provides details about NUMA and CPU pinning. The numatune command provides details about which NUMA node the virtual machine is using.

8.4. Live Migrate a Virtual Machine

Live migration moves a virtual machine from a source Compute node to a destination Compute node with a minimal amount of downtime. However, live migration might not be appropriate for all virtual machines. See Migration Constraints for additional details.

Procedure

  1. To live migrate a virtual machine, specify the virtual machine and the destination Compute node: 

    $ openstack server migrate [vm] --live [dest] --wait

    Replace [vm] with the name or ID of the virtual machine. Replace [dest] with the hostname of the destination Compute node. Specify the --block-migration flag if migrating a locally stored volume.

  2. Wait for migration to complete. See Check Migration Status to check the status of the migration.

  3. Confirm the migration was successful: 

    $ openstack server list --host [dest] --all-projects

    Replace [dest] with the hostname of the destination Compute node.

  4. For virtual machines using NUMA, CPU-pinning or DPDK, consider retrieving information about the NUMA topology from a Compute node to compare it with NUMA topology retrieved during the pre-migration procedure. 

    $ ssh root@overcloud-compute-n
# virsh vcpuinfo [vm]
# virsh numatune [vm]
# exit

    Replace overcloud-compute-n with the host name of the Compute node. Replace [vm] with the name of the virtual machine. Comparing the NUMA topologies of the source and destination Compute nodes helps to ensure that the source and destination Compute nodes use the same NUMA topology.

  5. Repeat this procedure for each additional virtual machine that you intend to migrate.

When you have finished migrating the virtual machines, proceed to the Post-migration Procedures.

8.5. Cold Migrate a Virtual Machine

Cold migrating a virtual machine involves stopping the virtual machine and moving it to another Compute node. Cold migration facilitates migration scenarios that live migrating cannot facilitate, such as migrating virtual machines using PCI passthrough or Single-Root Input/Output Virtualization (SR-IOV). The Scheduler automatically selects the destination Compute node. See Migration Constraints for additional details.

Procedure

  1. To migrate a virtual machine, specify the virtual machine. 

    $ openstack server migrate [vm] --wait

    Replace [vm] with the virtual machine ID. Specify the --block-migration flag if migrating a locally stored volume.

  2. Wait for migration to complete. See Check Migration Status to check the status of the migration.

  3. Confirm the migration was successful. 

    $ openstack server list --all-projects

When you have finished migrating virtual machines, proceed to the Post-migration Procedures.

8.6. Check Migration Status

Migration involves numerous state transitions before migration is complete. During a healthy migration, the migration state typically transitions as follows:

  1. Queued: nova accepted the request to migrate a virtual machine and migration is pending.
  2. Preparing: nova is preparing to migrate the virtual machine.
  3. Running: nova is in the process of migrating the virtual machine.
  4. Post-migrating: nova has built the virtual machine on the destination Compute node and is freeing up resources on the source Compute node.
  5. Completed: nova has completed migrating the virtual machine and finished freeing up resources on the source Compute node.

Procedure

  1. Retrieve the list of migrations for the virtual machine. 

    $ nova server-migration-list [vm]

    Replace [vm] with the virtual machine name or ID.

  2. Show the status of the migration. 

    $ nova server-migration-show [vm] [migration]

    Replace [vm] with the virtual machine name or ID. Replace [migration] with the ID of the migration.

Sometimes virtual machine migration can take a long time or encounter errors. See Section 8.8, “Troubleshooting Migration” for details.

8.7. Post-migration Procedures

After migrating one or more virtual machines, review the following procedures and execute them as appropriate.

Post-Migration Procedure for Compute Node Maintenance

If you previously shut down the source Compute node for maintenance and maintenance is complete, you may re-enable the source Compute node from the undercloud to ensure that the scheduler can assign new virtual machines to the source Compute node. 

$ source ~/overcloudrc
$ openstack compute service set [source] nova-compute --enable

Replace [source] with the host name of the source Compute node.

Post-Migration Procedure for NUMA, CPU-pinned or DPDK Instances

After migrating virtual machines that use NUMA, CPU-pinning or DPDK, you may re-enable the destination Compute node from the undercloud. 

$ source ~/overcloudrc
$ openstack compute service set [dest] nova-compute --enable

Replace [dest] with the host name of the destination Compute node.

8.8. Troubleshooting Migration

There are several issues that can arise during virtual machine migration:

  1. The migration process encounters errors.
  2. The migration process never ends.
  3. Virtual machine performance degrades after migration

Errors During Migration

The following issues can send the migration operation into an error state:

  1. Running a cluster with different versions of OpenStack.
  2. Specifying a virtual machine ID that cannot be found.
  3. The virtual machine you are trying to migrate is in an error state.
  4. The Compute service is shutting down.
  5. A race condition occurs.
  6. Live migration enters a failed state.

When live migration enters a failed state, it is typically followed by an error state. The following common issues can cause a failed state:

  1. A destination Compute host is not available.
  2. A scheduler exception occurs.
  3. The rebuild process fails due to insufficient computing resources.
  4. A server group check fails.
  5. The virtual machine on the source Compute node gets deleted before migration to the destination Compute node is complete.

Never-ending Live Migration

Live migration can fail to complete in a timely manner, which leaves migration in a perpetual running state. A common reason for a live migration that never completes is that client requests to the virtual machine running on the source Compute node create changes that occur faster than nova can replicate them to the destination Compute node.

There are a few ways to address this situation:

  1. Abort the live migration.
  2. Force the live migration to complete.

Aborting Live Migration

If the virtual machine state changes faster than the migration procedure can copy it to the destination node and you do not want to temporarily suspend the virtual machine’s operations, you can abort the live migration procedure.

  1. Retrieve the list of migrations for the virtual machine: 

    $ nova server-migration-list [vm]

    Replace [vm] with the virtual machine name or ID.

  2. Abort the live migration: 

    $ nova live-migration-abort [vm] [migration]

    Replace [vm] with the virtual machine name or ID, and [migration] with the ID of the migration.

Forcing Live Migration to Complete

If the virtual machine state changes faster than the migration procedure can copy it to the destination node and you want to temporarily suspend the virtual machine’s operations to force migration to complete, you can force the live migration procedure to complete.

Important

Forcing live migration to complete might lead to perceptible downtime.

  1. Retrieve the list of migrations for the virtual machine: 

    $ nova server-migration-list [vm]

    Replace [vm] with the virtual machine name or ID.

  2. Force the live migration to complete: 

    $ nova live-migration-force-complete [vm] [migration]

    Replace [vm] with the virtual machine name or ID. Replace [migration] with the ID of the migration.

Virtual Machine Performance Degrades After Migration

For VMs using a NUMA topology, the source and destination Compute nodes must have the same NUMA topology and configuration. The destination Compute node’s NUMA topology must have sufficient resources available. If the NUMA configuration between the source and destination Compute nodes is not the same, it is possible that live migration succeeds while the virtual machine performance degrades. For example, if the source Compute node maps NIC 1 to NUMA node 0, but the destination Compute node maps NIC 1 to NUMA node 5, after migration the virtual machine might route network traffic from a first CPU across the bus to a second CPU with NUMA node 5 to route traffic to NIC 1—​resulting in expected behavior, but degraded performance. Similarly, if NUMA node 0 on the source Compute node has sufficient available CPU and RAM, but NUMA node 0 on the destination Compute node already has virtual machines using some of the resources, the virtual machine might run properly but suffer performance degradation. See Section 8.2, “Migration Constraints” for additional details.