1. Install the Migration Toolkit for Containers Operator on all clusters.

   You can install the Migration Toolkit for Containers Operator in a restricted environment with limited or no internet access. The source and target clusters must have network access to each other and to a mirror registry.

2. Configure the replication repository, an intermediate object storage that MTC uses to migrate data.

   The source and target clusters must have network access to the replication repository during migration. In a restricted environment, you can use an internally hosted S3 storage repository. If you are using a proxy server, you must configure it to allow network traffic between the replication repository and the clusters.

3. Add the source cluster to the MTC web console.
4. Add the replication repository to the MTC web console.

5. Create a migration plan, with one of the following data migration options:

   • Copy: MTC copies the data from the source cluster to the replication repository, and from the replication repository to the target cluster.

     

   • Move: MTC unmounts a remote volume, for example, NFS, from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.

     Note

     Although the replication repository does not appear in this diagram, it is required for migration.

     

6. Run the migration plan, with one of the following options:

   • Stage (optional) copies data to the target cluster without stopping the application.

     Staging can be run multiple times so that most of the data is copied to the target before migration. This minimizes the duration of the migration and application downtime.

   • Migrate stops the application on the source cluster and recreates its resources on the target cluster. Optionally, you can migrate the workload without stopping the application.

Procedure

1. Log in to the OpenShift Container Platform cluster on which you have installed MTC.

2. Obtain the MTC web console URL by entering the following command:

   $ oc get -n openshift-migration route/migration -o go-template='https://{{ .spec.host }}'

   The output resembles the following: https://migration-openshift-migration.apps.cluster.openshift.com.

3. Launch a browser and navigate to the MTC web console.

   Note

   If you try to access the MTC web console immediately after installing the Migration Toolkit for Containers Operator, the console might not load because the Operator is still configuring the cluster. Wait a few minutes and retry.

4. If you are using self-signed CA certificates, you will be prompted to accept the CA certificate of the source cluster API server. The web page guides you through the process of accepting the remaining certificates.
5. Log in with your OpenShift Container Platform username and password.

Procedure

1. In the OpenShift Container Platform console, navigate to Operators → Installed Operators.
2. Click Migration Toolkit for Containers Operator.
3. In the Subscription tab, change the Approval option to Automatic.

4. Optional: Edit the Channel.

   Updating the subscription deploys the updated Migration Toolkit for Containers Operator and updates the MTC components.

5. Under Provided APIs, locate the Migration Controller tile, and click Create Instance.

6. If you are upgrading MTC on a 4.1 source cluster, update the migration_controller and migration_ui parameters and add the deprecated_cors_configuration parameter to the migration_controller manifest:

   spec:
   ...
     migration_controller: false
     migration_ui: false
     deprecated_cors_configuration: true

   Note

   You do not need to update the manifest of the target cluster.

7. Click Create.
8. Click Workloads → Pods to verify that the MTC pods are running.

Procedure

1. Log in to the target cluster.

2. Get the MigPlan CR:

   $ oc get migplan <migplan> -o yaml -n openshift-migration

3. Change the following parameter values to true and save the file:

   ...
   spec:
     indirectImageMigration: true
     indirectVolumeMigration: true

4. Apply the changes:

   $ oc replace -f <migplan>.yaml -n openshift-migration

5. Verify the changes by viewing the updated MigPlan CR:

   $ oc get migplan <migplan> -o yaml -n openshift-migration

Procedure

1. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket <bucket_name> \ 1
       --region <bucket_region> 2

   1

   Specify your S3 bucket name.

   2

   Specify your S3 bucket region, for example, us-east-1.

2. Create the IAM user velero:

   $ aws iam create-user --user-name velero

3. Create an EC2 EBS snapshot policy:

   $ cat > velero-ec2-snapshot-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           }
       ]
   }
   EOF

4. Create an AWS S3 access policy for one or for all S3 buckets:

   $ cat > velero-s3-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>/*" 1
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>" 2
               ]
           }
       ]
   }
   EOF

   1 2

   To grant access to a single S3 bucket, specify the bucket name. To grant access to all AWS S3 buckets, specify * instead of a bucket name as in the following example:

   Example output

   "Resource": [
       "arn:aws:s3:::*"

5. Attach the EC2 EBS policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-ebs \
     --policy-document file://velero-ec2-snapshot-policy.json

6. Attach the AWS S3 policy to velero:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero-s3 \
     --policy-document file://velero-s3-policy.json

7. Create an access key for velero:

   $ aws iam create-access-key --user-name velero
   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, 1
           "AccessKeyId": <AWS_ACCESS_KEY_ID> 2
       }
   }

   1 2

   Record the AWS_SECRET_ACCESS_KEY and the AWS_ACCESS_KEY_ID for adding the AWS repository to the MTC web console.

Procedure

1. Log in to gsutil:

   $ gsutil init

   Example output

   Welcome! This command will take you through the configuration of gcloud.
   
   Your current configuration has been set to: [default]
   
   To continue, you must login. Would you like to login (Y/n)?

2. Set the BUCKET variable:

   $ BUCKET=<bucket_name> 1

   1

   Specify your bucket name.

3. Create a storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a velero IAM service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero Storage"

6. Create the SERVICE_ACCOUNT_EMAIL variable:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
     --filter="displayName:Velero Storage" \
     --format 'value(email)')

7. Create the ROLE_PERMISSIONS variable:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
   )

8. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

9. Add IAM policy binding to the project:

   $ gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
       --role projects/$PROJECT_ID/roles/velero.server

10. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

11. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
      --iam-account $SERVICE_ACCOUNT_EMAIL

Procedure

1. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

2. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location <CentralUS> 1

   1

   Specify your location.

3. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID=velerobackups

4. Create an Azure storage account:

   $ az storage account create \
     --name $AZURE_STORAGE_ACCOUNT_ID \
     --resource-group $AZURE_RESOURCE_GROUP \
     --sku Standard_GRS \
     --encryption-services blob \
     --https-only true \
     --kind BlobStorage \
     --access-tier Hot

5. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

6. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

7. Create a service principal and credentials for velero:

   $ AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv` \
     AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv` \
     AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role "Contributor" --query 'password' -o tsv` \
     AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

8. Save the service principal credentials in the credentials-velero file:

   $ cat << EOF  > ./credentials-velero
   AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
   AZURE_TENANT_ID=${AZURE_TENANT_ID}
   AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
   AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
   AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
   AZURE_CLOUD_NAME=AzurePublicCloud
   EOF

Procedure

1. Log in to the cluster.

2. Obtain the service account token:

   $ oc sa get-token migration-controller -n openshift-migration

   Example output

   eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaWciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWlnLXRva2VuLWs4dDJyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1pZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE1YjFiYWMwLWMxYmYtMTFlOS05Y2NiLTAyOWRmODYwYjMwOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptaWc6bWlnIn0.xqeeAINK7UXpdRqAtOj70qhBJPeMwmgLomV9iFxr5RoqUgKchZRG2J2rkqmPm6vr7K-cm7ibD1IBpdQJCcVDuoHYsFgV4mp9vgOfn9osSDp2TGikwNz4Az95e81xnjVUmzh-NjDsEpw71DH92iHV_xt2sTwtzftS49LpPW2LjrV0evtNBP_t_RfskdArt5VSv25eORl7zScqfe1CiMkcVbf2UqACQjo3LbkpfN26HAioO2oH0ECPiRzT0Xyh-KwFutJLS9Xgghyw-LD9kPKcE_xbbJ9Y4Rqajh7WdPYuB0Jd9DPVrslmzK-F6cgHHYoZEv0SvLQi-PO0rpDrcjOEQQ

3. In the MTC web console, click Clusters.
4. Click Add cluster.

5. Fill in the following fields:

   • Cluster name: May contain lower-case letters (a-z) and numbers (0-9). Must not contain spaces or international characters.
   • Url: URL of the cluster’s API server, for example, https://<master1.example.com>:8443.
   • Service account token: String that you obtained from the source cluster.

   • Exposed route to image registry: Optional. You can specify a route to the image registry of your source cluster to enable direct migration for images, for example, docker-registry-default.apps.cluster.com.

     Direct migration is much faster than migration with a replication repository.

   • Azure cluster: Optional. Select it if you are using Azure snapshots to copy your data.
   • Azure resource group: This field appears if Azure cluster is checked.
   • If you use a custom CA bundle, click Browse and browse to the CA bundle file.

6. Click Add cluster.

   The cluster appears in the Clusters list.

Procedure

1. In the MTC web console, click Replication repositories.
2. Click Add repository.

3. Select a Storage provider type and fill in the following fields:

   • AWS for AWS S3, MCG, and generic S3 providers:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • S3 bucket name: Specify the name of the S3 bucket you created.
     • S3 bucket region: Specify the S3 bucket region. Required for AWS S3. Optional for other S3 providers.
     • S3 endpoint: Specify the URL of the S3 service, not the bucket, for example, https://<s3-storage.apps.cluster.com>. Required for a generic S3 provider. You must use the https:// prefix.
     • S3 provider access key: Specify the <AWS_SECRET_ACCESS_KEY> for AWS or the S3 provider access key for MCG.
     • S3 provider secret access key: Specify the <AWS_ACCESS_KEY_ID> for AWS or the S3 provider secret access key for MCG.
     • Require SSL verification: Clear this check box if you are using a generic S3 provider.
     • If you use a custom CA bundle, click Browse and browse to the Base64-encoded CA bundle file.

   • GCP:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • GCP bucket name: Specify the name of the GCP bucket.
     • GCP credential JSON blob: Specify the string in the credentials-velero file.

   • Azure:

     • Replication repository name: Specify the replication repository name in the MTC web console.
     • Azure resource group: Specify the resource group of the Azure Blob storage.
     • Azure storage account name: Specify the Azure Blob storage account name.
     • Azure credentials - INI file contents: Specify the string in the credentials-velero file.
4. Click Add repository and wait for connection validation.

5. Click Close.

   The new repository appears in the Replication repositories list.

Procedure

1. In the MTC web console, click Migration plans.
2. Click Add migration plan to launch the Migration Plan wizard.
3. In the General screen, enter the Plan name.
4. Select a source cluster, a target cluster, and a replication repository and then click Next.
5. In the Namespaces screen, select the projects to be migrated and then click Next.

6. In the Persistent volumes screen, click a Migration type for each PV:

   • The Copy option copies the data from the PV of a source cluster to the replication repository and then restores it on a newly created PV, with similar characteristics, in the target cluster.

     If you specified a route to an image registry when you added the source cluster to the web console, you can migrate images directly from the source cluster to the target cluster.

   • The Move option unmounts a remote volume, for example, NFS, from the source cluster, creates a PV resource on the target cluster pointing to the remote volume, and then mounts the remote volume on the target cluster. Applications running on the target cluster use the same remote volume that the source cluster was using. The remote volume must be accessible to the source and target clusters.
7. Click Next.

8. In the Copy options screen, click a Copy method for each PV:

   • The Snapshot copy option backs up and restores the disk using the cloud provider’s snapshot functionality. Copying snapshots is faster than copying the file system.

     Note

     The storage and clusters must be in the same region and the storage classes must be compatible.

   • The Filesystem copy option backs up the files on the source cluster and restores them on the target cluster.
9. Select Verify copy if you want to verify data migrated with Filesystem copy. Data is verified by generating a checksum for each source file and checking the checksum after restoration. This option significantly reduces performance.

10. Select a Target storage class for each PV.

    You can change the storage class of data migrated with Filesystem copy.

11. Click Next.

12. In the Migration options screen, the Direct image migration option is selected if you specified an exposed image registry route for the source cluster. The Direct PV migration option is selected if you are migrating data with Filesystem copy.

    The direct migration options copy images and files directly from the source cluster to the target cluster. This option is much faster than copying images and files from the source cluster to the replication repository and then from the replication repository to the target cluster.

13. Click Next.
14. In the Hooks screen, click Add Hook to add a hook to the migration plan.
15. Enter the hook name.
16. If your hook is an Ansible playbook, click Browse to upload the playbook and update the Ansible runtime image field if you are using a custom Ansible image.
17. If your hook is not an Ansible playbook, click Custom container image and specify the image name and path.
18. Click Source cluster or Target cluster on which the hook should run.
19. Enter the Service account name and the Service account namespace of the cluster.

20. Select the migration step when the hook should run:

    • PreBackup: Before backup tasks are started on the source cluster
    • PostBackup: After backup tasks are complete on the source cluster
    • PreRestore: Before restore tasks are started on the target cluster
    • PostRestore: After restore tasks are complete on the target cluster

21. Click Add Hook and then click Close.

    You can add up to four hooks to a single migration plan. Each hook runs during a different migration step.

22. Click Finish and then click Close.

    The migration plan is displayed in the Migration plans list.

Procedure

1. Log in to the source cluster.

2. Delete old images:

   $ oc adm prune images

3. Log in to the MTC web console and click Migration plans.

4. Click the Options menu next to a migration plan and select Stage to copy data from the source cluster to the target cluster without stopping the application.

   You can run Stage multiple times to reduce the actual migration time.

5. When you are ready to migrate the application workload, the Options menu beside a migration plan and select Migrate.
6. Optional: In the Migrate window, you can select Do not stop applications on the source cluster during migration.
7. Click Migrate.

8. When the migration is complete, verify that the application migrated successfully in the OpenShift Container Platform web console:

   1. Click Home → Projects.
   2. Click the migrated project to view its status.
   3. In the Routes section, click Location to verify that the application is functioning, if applicable.
   4. Click Workloads → Pods to verify that the pods are running in the migrated namespace.
   5. Click Storage → Persistent volumes to verify that the migrated persistent volume is correctly provisioned.

Procedure

1. List the MigMigration CRs in the openshift-migration namespace:

   $ oc get migmigration -n openshift-migration

   Example output

   NAME                                   AGE
   88435fe0-c9f8-11e9-85e6-5d593ce65e10   6m42s

2. Inspect the MigMigration CR:

   $ oc describe migmigration 88435fe0-c9f8-11e9-85e6-5d593ce65e10 -n openshift-migration

   The output is similar to the following examples.

Procedure

1. Get the mig-log-reader pod:

   $ oc -n openshift-migration get pods | grep log

2. Enter the following command to display a single migration log:

   $ oc -n openshift-migration logs -f <mig-log-reader-pod> -c color 1

   1

   The -c plain option displays the log without colors.

Procedure

1. In the MTC console, click Migration plans to view the list of migration plans.
2. Click the Options menu of a specific migration plan and select Logs.

3. Click Download Logs to download the logs of the MigrationController, Velero, and Restic pods for all clusters.

   You can download a single log by selecting the cluster, log source, and pod source, and then clicking Download Selected.

   You can access a pod log from the CLI by using the oc logs command:

   $ oc logs <pod-name> -f -n openshift-migration 1

   1

   Specify the pod name.

Procedure

1. Check the status of the MigMigration CR:

   $ oc describe migmigration <pod_name> -n openshift-migration

   The output includes the following status message:

   Example output

   ...
   Some or all transfer pods are not running for more than 10 mins on destination cluster
   ...

2. On the source cluster, obtain the details of a migrated namespace:

   $ oc get namespace <namespace> -o yaml 1

   1

   Specify the migrated namespace.

3. On the target cluster, edit the migrated namespace:

   $ oc edit namespace <namespace>

4. Add missing openshift.io/node-selector annotations to the migrated namespace as in the following example:

   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       openshift.io/node-selector: "region=east"
   ...

5. Run the migration plan again.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the must-gather command:

   $ oc adm must-gather --image=openshift-migration-must-gather-rhel8:v1.4.0

3. Remove authentication keys and other sensitive information.

4. Create an archive file containing the contents of the must-gather data directory:

   $ tar cvaf must-gather.tar.gz must-gather.local.<uid>/

5. Upload the compressed file as an attachment to your customer support case.