Chapter 12. Live migration

12.1. Virtual machine live migration

12.1.1. About live migration

Live migration is the process of moving a running virtual machine instance (VMI) to another node in the cluster without interrupting the virtual workload or access. If a VMI uses the LiveMigrate eviction strategy, it automatically migrates when the node that the VMI runs on is placed into maintenance mode. You can also manually start live migration by selecting a VMI to migrate.

You can use live migration if the following conditions are met:

  • Shared storage with ReadWriteMany (RWX) access mode.
  • Sufficient RAM and network bandwidth.
  • If the virtual machine uses a host model CPU, the nodes must support the virtual machine’s host model CPU.

By default, live migration traffic is encrypted using Transport Layer Security (TLS).

12.1.2. Additional resources

12.2. Live migration limits and timeouts

Apply live migration limits and timeouts so that migration processes do not overwhelm the cluster. Configure these settings by editing the HyperConverged custom resource (CR).

12.2.1. Configuring live migration limits and timeouts

Configure live migration limits and timeouts for the cluster by updating the HyperConverged custom resource (CR), which is located in the openshift-cnv namespace.

Procedure

  • Edit the HyperConverged CR and add the necessary live migration parameters.

    $ oc edit hco -n openshift-cnv kubevirt-hyperconverged

    Example configuration file

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
      namespace: openshift-cnv
    spec:
      liveMigrationConfig: 1
        bandwidthPerMigration: 64Mi
        completionTimeoutPerGiB: 800
        parallelMigrationsPerCluster: 5
        parallelOutboundMigrationsPerNode: 2
        progressTimeout: 150

    1
    In this example, the spec.liveMigrationConfig array contains the default values for each field.
    Note

    You can restore the default value for any spec.liveMigrationConfig field by deleting that key/value pair and saving the file. For example, delete progressTimeout: <value> to restore the default progressTimeout: 150.

12.2.2. Cluster-wide live migration limits and timeouts

Table 12.1. Migration parameters

ParameterDescriptionDefault

parallelMigrationsPerCluster

Number of migrations running in parallel in the cluster.

5

parallelOutboundMigrationsPerNode

Maximum number of outbound migrations per node.

2

bandwidthPerMigration

Bandwidth limit of each migration, where the value is the quantity of bytes per second. For example, a value of 2048Mi means 2048 MiB/s.

0 [1]

completionTimeoutPerGiB

The migration is canceled if it has not completed in this time, in seconds per GiB of memory. For example, a virtual machine instance with 6GiB memory times out if it has not completed migration in 4800 seconds. If the Migration Method is BlockMigration, the size of the migrating disks is included in the calculation.

800

progressTimeout

The migration is canceled if memory copy fails to make progress in this time, in seconds.

150

  1. The default value of 0 is unlimited.

12.3. Migrating a virtual machine instance to another node

Manually initiate a live migration of a virtual machine instance to another node using either the web console or the CLI.

Note

If a virtual machine uses a host model CPU, you can perform live migration of that virtual machine only between nodes that support its host CPU model.

12.3.1. Initiating live migration of a virtual machine instance in the web console

Migrate a running virtual machine instance to a different node in the cluster.

Note

The Migrate action is visible to all users but only admin users can initiate a virtual machine migration.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationVirtualMachines from the side menu.
  2. You can initiate the migration from this page, which makes it easier to perform actions on multiple virtual machines on the same page, or from the VirtualMachine details page where you can view comprehensive details of the selected virtual machine:

    • Click the Options menu kebab next to the virtual machine and select Migrate.
    • Click the virtual machine name to open the VirtualMachine details page and click ActionsMigrate.
  3. Click Migrate to migrate the virtual machine to another node.

12.3.1.1. Monitoring live migration by using the web console

You can monitor the progress of all live migrations on the OverviewMigrations tab in the web console.

You can view the migration metrics of a virtual machine on the VirtualMachine detailsMetrics tab in the web console.

12.3.2. Initiating live migration of a virtual machine instance in the CLI

Initiate a live migration of a running virtual machine instance by creating a VirtualMachineInstanceMigration object in the cluster and referencing the name of the virtual machine instance.

Procedure

  1. Create a VirtualMachineInstanceMigration configuration file for the virtual machine instance to migrate. For example, vmi-migrate.yaml:

    apiVersion: kubevirt.io/v1
    kind: VirtualMachineInstanceMigration
    metadata:
      name: migration-job
    spec:
      vmiName: vmi-fedora
  2. Create the object in the cluster by running the following command:

    $ oc create -f vmi-migrate.yaml

The VirtualMachineInstanceMigration object triggers a live migration of the virtual machine instance. This object exists in the cluster for as long as the virtual machine instance is running, unless manually deleted.

12.3.2.1. Monitoring live migration of a virtual machine instance in the CLI

The status of the virtual machine migration is stored in the Status component of the VirtualMachineInstance configuration.

Procedure

  • Use the oc describe command on the migrating virtual machine instance:

    $ oc describe vmi vmi-fedora

    Example output

    # ...
    Status:
      Conditions:
        Last Probe Time:       <nil>
        Last Transition Time:  <nil>
        Status:                True
        Type:                  LiveMigratable
      Migration Method:  LiveMigration
      Migration State:
        Completed:                    true
        End Timestamp:                2018-12-24T06:19:42Z
        Migration UID:                d78c8962-0743-11e9-a540-fa163e0c69f1
        Source Node:                  node2.example.com
        Start Timestamp:              2018-12-24T06:19:35Z
        Target Node:                  node1.example.com
        Target Node Address:          10.9.0.18:43891
        Target Node Domain Detected:  true

12.3.3. Additional resources

12.4. Migrating a virtual machine over a dedicated additional network

You can configure a dedicated Multus network for live migration. A dedicated network minimizes the effects of network saturation on tenant workloads during live migration.

12.4.1. Configuring a dedicated secondary network for virtual machine live migration

To configure a dedicated secondary network for live migration, you must first create a bridge network attachment definition for the openshift-cnv namespace by using the CLI. Then, add the name of the NetworkAttachmentDefinition object to the HyperConverged custom resource (CR).

Prerequisites

  • You installed the OpenShift CLI (oc).
  • You logged in to the cluster as a user with the cluster-admin role.
  • The Multus Container Network Interface (CNI) plugin is installed on the cluster.
  • Every node on the cluster has at least two Network Interface Cards (NICs), and the NICs to be used for live migration are connected to the same VLAN.
  • The virtual machine (VM) is running with the LiveMigrate eviction strategy.

Procedure

  1. Create a NetworkAttachmentDefinition manifest.

    Example configuration file

    apiVersion: "k8s.cni.cncf.io/v1"
    kind: NetworkAttachmentDefinition
    metadata:
      name: my-secondary-network 1
      namespace: openshift-cnv 2
    spec:
      config: '{
        "cniVersion": "0.3.1",
        "name": "migration-bridge",
        "type": "macvlan",
        "master": "eth1", 3
        "mode": "bridge",
        "ipam": {
          "type": "whereabouts", 4
          "range": "10.200.5.0/24" 5
        }
      }'

    1
    The name of the NetworkAttachmentDefinition object.
    2
    The namespace where the NetworkAttachmentDefinition object resides. This must be openshift-cnv.
    3
    The name of the NIC to be used for live migration.
    4
    The name of the CNI plugin that provides the network for this network attachment definition.
    5
    The IP address range for the secondary network. This range must not have any overlap with the IP addresses of the main network.
  2. Open the HyperConverged CR in your default editor by running the following command:

    oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
  3. Add the name of the NetworkAttachmentDefinition object to the spec.liveMigrationConfig stanza of the HyperConverged CR. For example:

    Example configuration file

    apiVersion: hco.kubevirt.io/v1beta1
    kind: HyperConverged
    metadata:
      name: kubevirt-hyperconverged
    spec:
      liveMigrationConfig:
        completionTimeoutPerGiB: 800
        network: my-secondary-network  1
        parallelMigrationsPerCluster: 5
        parallelOutboundMigrationsPerNode: 2
        progressTimeout: 150
    # ...

    1
    The name of the Multus NetworkAttachmentDefinition object to be used for live migrations.
  4. Save your changes and exit the editor. The virt-handler pods restart and connect to the secondary network.

Verification

  • When the node that the virtual machine runs on is placed into maintenance mode, the VM automatically migrates to another node in the cluster. You can verify that the migration occurred over the secondary network and not the default pod network by checking the target IP address in the virtual machine instance (VMI) metadata.

    oc get vmi <vmi_name> -o jsonpath='{.status.migrationState.targetNodeAddress}'

12.4.2. Selecting a dedicated network by using the web console

You can select a dedicated network for live migration by using the OpenShift Container Platform web console.

Prerequisites

  • You configured a Multus network for live migration.

Procedure

  1. Navigate to Virtualization > Overview in the OpenShift Container Platform web console.
  2. Click the Settings tab and then click Live migration.
  3. Select the network from the Live migration network list.

12.4.3. Additional resources

12.5. Cancelling the live migration of a virtual machine instance

Cancel the live migration so that the virtual machine instance remains on the original node.

You can cancel a live migration from either the web console or the CLI.

12.5.1. Cancelling live migration of a virtual machine instance in the web console

You can cancel the live migration of a virtual machine instance in the web console.

Procedure

  1. In the OpenShift Container Platform console, click VirtualizationVirtualMachines from the side menu.
  2. Click the Options menu kebab beside a virtual machine and select Cancel Migration.

12.5.2. Cancelling live migration of a virtual machine instance in the CLI

Cancel the live migration of a virtual machine instance by deleting the VirtualMachineInstanceMigration object associated with the migration.

Procedure

  • Delete the VirtualMachineInstanceMigration object that triggered the live migration, migration-job in this example:

    $ oc delete vmim migration-job

12.6. Configuring virtual machine eviction strategy

The LiveMigrate eviction strategy ensures that a virtual machine instance is not interrupted if the node is placed into maintenance or drained. Virtual machines instances with this eviction strategy will be live migrated to another node.

12.6.1. Configuring custom virtual machines with the LiveMigration eviction strategy

You only need to configure the LiveMigration eviction strategy on custom virtual machines. Common templates have this eviction strategy configured by default.

Procedure

  1. Add the evictionStrategy: LiveMigrate option to the spec.template.spec section in the virtual machine configuration file. This example uses oc edit to update the relevant snippet of the VirtualMachine configuration file:

    $ oc edit vm <custom-vm> -n <my-namespace>
    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      name: custom-vm
    spec:
      template:
        spec:
          evictionStrategy: LiveMigrate
    # ...
  2. Restart the virtual machine for the update to take effect:

    $ virtctl restart <custom-vm> -n <my-namespace>

12.7. Configuring live migration policies

You can define different migration configurations for specified groups of virtual machine instances (VMIs) by using a live migration policy.

Important

Live migration policy is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

To configure a live migration policy by using the web console, see the MigrationPolicies page documentation.

12.7.1. Configuring a live migration policy from the command line

Use the MigrationPolicy custom resource definition (CRD) to define migration policies for one or more groups of selected virtual machine instances (VMIs).

You can specify groups of VMIs by using any combination of the following:

  • Virtual machine instance labels such as size, os, gpu, and other VMI labels.
  • Namespace labels such as priority, bandwidth, hpc-workload, and other namespace labels.

For the policy to apply to a specific group of VMIs, all labels on the group of VMIs must match the labels in the policy.

Note

If multiple live migration policies apply to a VMI, the policy with the highest number of matching labels takes precedence. If multiple policies meet this criteria, the policies are sorted by lexicographic order of the matching labels keys, and the first one in that order takes precedence.

Procedure

  1. Create a MigrationPolicy CRD for your specified group of VMIs. The following example YAML configures a group with the labels hpc-workloads:true, xyz-workloads-type: "", workload-type: db, and operating-system: "":

    apiVersion: migrations.kubevirt.io/v1beta1
    kind: MigrationPolicy
    metadata:
      name: my-awesome-policy
    spec:
      # Migration Configuration
      allowAutoConverge: true
      bandwidthPerMigration: 217Ki
      completionTimeoutPerGiB: 23
      allowPostCopy: false
    
      # Matching to VMIs
      selectors:
        namespaceSelector: 1
          hpc-workloads: "True"
          xyz-workloads-type: ""
        virtualMachineInstanceSelector: 2
          workload-type: "db"
          operating-system: ""
    1
    Use namespaceSelector to define a group of VMIs by using namespace labels.
    2
    Use virtualMachineInstanceSelector to define a group of VMIs by using VMI labels.