Red Hat Training

A Red Hat training course is available for Red Hat Virtualization

6.13. Migrating Virtual Machines Between Hosts

Live migration provides the ability to move a running virtual machine between physical hosts with no interruption to service. The virtual machine remains powered on and user applications continue to run while the virtual machine is relocated to a new physical host. In the background, the virtual machine's RAM is copied from the source host to the destination host. Storage and network connectivity are not altered.

6.13.1. Live Migration Prerequisites

Live migration is used to seamlessly move virtual machines to support a number of common maintenance tasks. Ensure that your Red Hat Virtualization environment is correctly configured to support live migration well in advance of using it.
At a minimum, for successful live migration of virtual machines to be possible:
  • The source and destination host should both be members of the same cluster, ensuring CPU compatibility between them.

    Note

    Live migrating virtual machines between different clusters is generally not recommended. The currently only supported use case is documented at https://access.redhat.com/articles/1390733.
  • The source and destination host must have a status of Up.
  • The source and destination host must have access to the same virtual networks and VLANs.
  • The source and destination host must have access to the data storage domain on which the virtual machine resides.
  • There must be enough CPU capacity on the destination host to support the virtual machine's requirements.
  • There must be enough RAM on the destination host that is not in use to support the virtual machine's requirements.
  • The migrating virtual machine must not have the cache!=none custom property set.
In addition, for best performance, the storage and management networks should be split to avoid network saturation. Virtual machine migration involves transferring large amounts of data between hosts.
Live migration is performed using the management network. Each live migration event is limited to a maximum transfer speed of 30 MBps, and the number of concurrent migrations supported is also limited by default. Despite these measures, concurrent migrations have the potential to saturate the management network. It is recommended that separate logical networks are created for storage, display, and virtual machine data to minimize the risk of network saturation.

Additional Prerequisites for Virtual Machines with SR-IOV-Enabled vNICs

Virtual machines with vNICs that are directly connected to a virtual function (VF) of an SR-IOV-enabled host NIC have additional requirements for successful migration:
  • There must be an available VF on the destination host.
  • The vNIC profile for the passthrough vNIC must have Passthrough and Migratable selected. See Enabling Passthrough on a vNIC Profile in the Administration Guide for more information.
  • The virtual machine must have a backup VirtIO vNIC, in addition to the passthrough vNIC, to maintain the virtual machine's network connection during migration.
  • Both vNICs must be added as slaves under an active-backup bond on the virtual machine, with the passthrough vNIC as the primary interface. See Configure Network Bonding in the Red Hat Enterprise Linux Networking Guide for more information.

6.13.2. Optimizing Live Migration

Live virtual machine migration can be a resource-intensive operation. The following two options can be set globally for every virtual machine in the environment, at the cluster level, or at the individual virtual machine level to optimize live migration.
The Auto Converge migrations option allows you to set whether auto-convergence is used during live migration of virtual machines. Large virtual machines with high workloads can dirty memory more quickly than the transfer rate achieved during live migration, and prevent the migration from converging. Auto-convergence capabilities in QEMU allow you to force convergence of virtual machine migrations. QEMU automatically detects a lack of convergence and triggers a throttle-down of the vCPUs on the virtual machine.
The Enable migration compression option allows you to set whether migration compression is used during live migration of the virtual machine. This feature uses Xor Binary Zero Run-Length-Encoding to reduce virtual machine downtime and total live migration time for virtual machines running memory write-intensive workloads or for any application with a sparse memory update pattern.
Both options are disabled globally by default.

Procedure 6.31. Configuring Auto-convergence and Migration Compression for Virtual Machine Migration

  1. Configure the optimization settings at the global level:
    1. Enable auto-convergence at the global level:
      # engine-config -s DefaultAutoConvergence=True
    2. Enable migration compression at the global level:
      # engine-config -s DefaultMigrationCompression=True
    3. Restart the ovirt-engine service to apply the changes:
      # systemctl restart ovirt-engine.service
  2. Configure the optimization settings at the cluster level:
    1. Select a cluster.
    2. Click Edit.
    3. Click the Migration Policy tab.
    4. From the Auto Converge migrations list, select Inherit from global setting, Auto Converge, or Don't Auto Converge.
    5. From the Enable migration compression list, select Inherit from global setting, Compress, or Don't Compress.
  3. Configure the optimization settings at the virtual machine level:
    1. Select a virtual machine.
    2. Click Edit.
    3. Click the Host tab.
    4. From the Auto Converge migrations list, select Inherit from cluster setting, Auto Converge, or Don't Auto Converge.
    5. From the Enable migration compression list, select Inherit from cluster setting, Compress, or Don't Compress.

6.13.3. Guest Agent Hooks

Hooks are scripts that trigger activity within a virtual machine when key events occur:
  • Before migration
  • After migration
  • Before hibernation
  • After hibernation
The hooks configuration base directory is /etc/ovirt-guest-agent/hooks.d on Linux systems and C:\Program Files\Redhat\RHEV\Drivers\Agent on Windows systems.
Each event has a corresponding subdirectory: before_migration and after_migration, before_hibernation and after_hibernation. All files or symbolic links in that directory will be executed.
The executing user on Linux systems is ovirtagent. If the script needs root permissions, the elevation must be executed by the creator of the hook script.
The executing user on Windows systems is the System Service user.

6.13.4. Automatic Virtual Machine Migration

Red Hat Virtualization Manager automatically initiates live migration of all virtual machines running on a host when the host is moved into maintenance mode. The destination host for each virtual machine is assessed as the virtual machine is migrated, in order to spread the load across the cluster.
The Manager automatically initiates live migration of virtual machines in order to maintain load balancing or power saving levels in line with scheduling policy. While no scheduling policy is defined by default, it is recommended that you specify the scheduling policy which best suits the needs of your environment. You can also disable automatic, or even manual, live migration of specific virtual machines where required.

6.13.5. Preventing Automatic Migration of a Virtual Machine

Red Hat Virtualization Manager allows you to disable automatic migration of virtual machines. You can also disable manual migration of virtual machines by setting the virtual machine to run only on a specific host.
The ability to disable automatic migration and require a virtual machine to run on a particular host is useful when using application high availability products, such as Red Hat High Availability or Cluster Suite.

Procedure 6.32. Preventing Automatic Migration of Virtual Machine

  1. Click the Virtual Machines tab and select a virtual machine.
  2. Click Edit.
    The Edit Virtual Machine Window

    Figure 6.11. The Edit Virtual Machine Window

  3. Click the Host tab.
  4. Use the Start Running On radio buttons to specify whether the virtual machine should run on any host in the cluster, or a specific host or group of hosts.

    Warning

    Explicitly assigning a virtual machine to one specific host and disabling migration is mutually exclusive with Red Hat Virtualization high availability. Virtual machines that are assigned to one specific host can only be made highly available using third party high availability products like Red Hat High Availability. This restriction does not apply to virtual machines that are assigned to multiple specific hosts.

    Important

    If the virtual machine has host devices directly attached to it, and a different host is specified, the host devices from the previous host will be automatically removed from the virtual machine.
  5. Select Allow manual migration only or Do not allow migration from the Migration Options drop-down list.
  6. Optionally, select the Use custom migration downtime check box and specify a value in milliseconds.
  7. Click OK.

6.13.6. Manually Migrating Virtual Machines

A running virtual machine can be live migrated to any host within its designated host cluster. Live migration of virtual machines does not cause any service interruption. Migrating virtual machines to a different host is especially useful if the load on a particular host is too high. For live migration prerequisites, see Section 6.13.1, “Live Migration Prerequisites”.

Note

When you place a host into maintenance mode, the virtual machines running on that host are automatically migrated to other hosts in the same cluster. You do not need to manually migrate these virtual machines.

Note

Live migrating virtual machines between different clusters is generally not recommended. The currently only supported use case is documented at https://access.redhat.com/articles/1390733.

Procedure 6.33. Manually Migrating Virtual Machines

  1. Click the Virtual Machines tab and select a running virtual machine.
  2. Click Migrate.
  3. Use the radio buttons to select whether to Select Host Automatically or to Select Destination Host, specifying the host using the drop-down list.

    Note

    When the Select Host Automatically option is selected, the system determines the host to which the virtual machine is migrated according to the load balancing and power management rules set up in the scheduling policy.
  4. Click OK.
During migration, progress is shown in the Migration progress bar. Once migration is complete the Host column will update to display the host the virtual machine has been migrated to.

6.13.7. Setting Migration Priority

Red Hat Virtualization Manager queues concurrent requests for migration of virtual machines off of a given host. The load balancing process runs every minute. Hosts already involved in a migration event are not included in the migration cycle until their migration event has completed. When there is a migration request in the queue and available hosts in the cluster to action it, a migration event is triggered in line with the load balancing policy for the cluster.
You can influence the ordering of the migration queue by setting the priority of each virtual machine; for example, setting mission critical virtual machines to migrate before others. Migrations will be ordered by priority; virtual machines with the highest priority will be migrated first.

Procedure 6.34. Setting Migration Priority

  1. Click the Virtual Machines tab and select a virtual machine.
  2. Click Edit.
  3. Select the High Availability tab.
  4. Select Low, Medium, or High from the Priority drop-down list.
  5. Click OK.

6.13.8. Canceling Ongoing Virtual Machine Migrations

A virtual machine migration is taking longer than you expected. You'd like to be sure where all virtual machines are running before you make any changes to your environment.

Procedure 6.35. Canceling Ongoing Virtual Machine Migrations

  1. Select the migrating virtual machine. It is displayed in the Virtual Machines resource tab with a status of Migrating from.
  2. Click Cancel Migration.
The virtual machine status returns from Migrating from to Up.

6.13.9. Event and Log Notification upon Automatic Migration of Highly Available Virtual Servers

When a virtual server is automatically migrated because of the high availability function, the details of an automatic migration are documented in the Events tab and in the engine log to aid in troubleshooting, as illustrated in the following examples:

Example 6.1. Notification in the Events Tab of the Web Admin Portal

Highly Available Virtual_Machine_Name failed. It will be restarted automatically.
Virtual_Machine_Name was restarted on Host Host_Name

Example 6.2. Notification in the Manager engine.log

This log can be found on the Red Hat Virtualization Manager at /var/log/ovirt-engine/engine.log:
Failed to start Highly Available VM. Attempting to restart. VM Name: Virtual_Machine_Name, VM Id:Virtual_Machine_ID_Number