Chapter 8. Troubleshooting

This section provides information for troubleshooting common migration issues.

8.1. Architecture

This section describes MTV custom resources, services, and workflows.

8.1.1. MTV custom resources and services

The Migration Toolkit for Virtualization (MTV) is provided as an OpenShift Container Platform Operator. It creates and manages the following custom resources (CRs) and services.

MTV custom resources

  • Provider CR stores attributes that enable MTV to connect to and interact with the source and target providers.
  • NetworkMapping CR maps the networks of the source and target providers.
  • StorageMapping CR maps the storage of the source and target providers.
  • Provisioner CR stores the configuration of the storage provisioners, such as supported volume and access modes.
  • Plan CR contains a list of VMs with the same migration parameters and associated network and storage mappings.

  • Migration CR runs a migration plan.

    Only one Migration CR per migration plan can run at a given time. You can create multiple Migration CRs for a single Plan CR.

MTV services

  • Provider Inventory service:

    • Connects to the source and target providers.
    • Maintains a local inventory for mappings and plans.
    • Stores VM configurations.
    • Runs the Validation service if a VM configuration change is detected.
  • Validation service checks the suitability of a VM for migration by applying rules.
Important

The Validation service is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

  • User Interface service:

    • Enables you to create and configure MTV CRs.
    • Displays the status of the CRs and the progress of a migration.

  • Migration Controller service orchestrates migrations.

    When you create a migration plan, the Migration Controller service validates the plan and adds a status label. If the plan fails validation, the plan status is Not ready and the plan cannot be used to perform a migration. If the plan passes validation, the plan status is Ready and it can be used to perform a migration. After a successful migration, the Migration Controller changes the plan status to Completed.

  • Virtual Machine Import Controller, Kubevirt Controller, and Containerized Data Import (CDI) Controller services handle most technical operations.

8.1.2. High-level migration workflow

The high-level workflow shows the migration process from the point of view of the user.

Figure 8.1. High-level workflow

The workflow describes the following steps:

  1. You create a source provider, a target provider, a network mapping, and a storage mapping.

  2. You create a migration plan that includes the following resources:

    • Source provider
    • Target provider
    • Network mapping
    • Storage mapping
    • One or more VMs
  3. You run a migration plan by creating a Migration CR that references the migration plan. If a migration is incomplete, you can run a migration plan multiple times until all VMs are migrated.
  4. For each VM in the migration plan, the Migration Controller creates a VirtualMachineImport CR and monitors its status. When all VMs have been migrated, the Migration Controller sets the status of the migration plan to Completed. The power state of a source VM is maintained after migration.

8.1.3. Detailed migration workflow

You can use the detailed migration workflow to troubleshoot a failed migration.

Figure 8.2. Detailed OpenShift Virtualization migration workflow

The workflow describes the following steps:

  1. When you run a migration plan, the Migration Controller creates a VirtualMachineImport custom resource (CR) for each source virtual machine (VM).
  2. The Virtual Machine Import Controller validates the VirtualMachineImport CR and generates a VirtualMachine CR.

  3. The Virtual Machine Import Controller retrieves the VM configuration, including network, storage, and metadata, linked in the VirtualMachineImport CR.



    For each VM disk:

  4. The Virtual Machine Import Controller creates a DataVolume CR as a wrapper for a Persistent Volume Claim (PVC) and annotations.


  5. The Containerized Data Importer (CDI) Controller creates a PVC. The Persistent Volume (PV) is dynamically provisioned by the StorageClass provisioner.


  6. The CDI Controller creates an Importer pod.

  7. For a VMware provider, the Importer pod connects to the VM disk by using the VMware Virtual Disk Development Kit (VDDK) SDK and streams the VM disk to the PV.

    After the VM disks are transferred:

  8. The Virtual Machine Import Controller creates a Conversion pod with the PVCs attached to it.

    The Conversion pod runs virt-v2v, which installs and configures device drivers on the PVCs of the target VM.

  9. The Virtual Machine Import Controller creates a VirtualMachineInstance CR.

  10. When the target VM is powered on, the KubeVirt Controller creates a VM pod.

    The VM pod runs QEMU-KVM with the PVCs attached as VM disks.

8.2. Error messages

This section describes error messages and how to resolve them.

warm import retry limit reached

The warm import retry limit reached error message is displayed during a warm migration if a VMware virtual machine (VM) has reached the maximum number (28) of changed block tracking (CBT) snapshots during the precopy stage. You must delete some of the CBT snapshots from the VM and restart the migration plan.

8.3. Using the must-gather tool

You can collect logs and information about MTV custom resources (CRs) by using the must-gather tool. You must attach a must-gather data file to all customer cases.

You can gather data for a specific namespace, migration plan, or virtual machine (VM) by using the filtering options.

Note

If you specify a non-existent resource in the filtered must-gather command, no archive file is created.

Prerequisites

Collecting logs and CR information

  1. Navigate to the directory where you want to store the must-gather data.

  2. Run the oc adm must-gather command: 

    $ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.1.0

    The data is saved as /must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.

  3. Optional: Run the oc adm must-gather command with the following options to gather filtered data:

    • Namespace: 

      $ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.1.0 \
  -- NS=<namespace> /usr/bin/targeted

    • Migration plan: 

      $ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.1.0 \
  -- PLAN=<migration_plan> /usr/bin/targeted

    • Virtual machine: 

      $ oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.1.0 \
  -- VM=<vm_id> NS=<namespace> /usr/bin/targeted 1
      1
      Specify the VM ID as it appears in the Plan CR.