Chapter 3. Live migration of images

As a storage administrator, you can live-migrate RBD images between different pools or even with the same pool, within the same storage cluster. You can migrate between different images formats and layouts and even from external data sources. When live migration is initiated, the source image is deep copied to the destination image, pulling all snapshot history while preserving the sparse allocation of data where possible.

Important

Currently, the krbd kernel module does not support live migration.

3.1. Prerequisites

  • A running Red Hat Ceph Storage cluster.

3.2. The live migration process

By default, during the live migration of the RBD images with the same storage cluster, the source image is marked read-only. All clients redirect the Input/Output (I/O) to the new target image. Additionally, this mode can preserve the link to the source image’s parent to preserve sparseness, or it can flatten the image during the migration to remove the dependency on the source image’s parent. You can use the live migration process in an import-only mode, where the source image remains unmodified. You can link the target image to an external data source, such as a backup file, HTTP(s) file, or an S3 object. The live migration copy process can safely run in the background while the new target image is being used.

The live migration process consists of three steps:

Prepare Migration: The first step is to create new target image and link the target image to the source image. If the import-only mode is not configured, the source image will also be linked to the target image and marked read-only. Attempts to read uninitialized data extents within the target image will internally redirect the read to the source image, and writes to uninitialized extents within the target image will internally deep copy, the overlapping source image extents to the target image.

Execute Migration: This is a background operation that deep-copies all initialized blocks from the source image to the target. You can run this step when clients are actively using the new target image.

Finish Migration: You can commit or abort the migration, once the background migration process is completed. Committing the migration removes the cross-links between the source and target images, and will remove the source image if not configured in the import-only mode. Aborting the migration remove the cross-links, and will remove the target image.

3.3. Formats

You can use the native format to describe a native RBD image within a Red Hat Ceph Storage cluster as the source image. The source-spec JSON document is encoded as:

Syntax

{
    "type": "native",
    "pool_name": "POOL_NAME",
    ["pool_id": "POOL_ID",] (optional, alternative to "POOL_NAME" key)
    ["pool_namespace": "POOL_NAMESPACE",] (optional)
    "image_name": "IMAGE_NAME>",
    ["image_id": "IMAGE_ID",] (optional, useful if image is in trash)
    "snap_name": "SNAP_NAME",
    ["snap_id": "SNAP_ID",] (optional, alternative to "SNAP_NAME" key)
}

Note that the native format does not include the stream object since it utilizes native Ceph operations. For example, to import from the image rbd/ns1/image1@snap1, the source-spec could be encoded as:

Example

{
"type": "native",
"pool_name": "rbd",
"pool_namespace": "ns1",
"image_name": "image1",
"snap_name": "snap1"
}

You can use the qcow format to describe a QEMU copy-on-write (QCOW) block device. Both the QCOW v1 and v2 formats are currently supported with the exception of advanced features such as compression, encryption, backing files, and external data files. You can link the qcow format data to any supported stream source:

Example

{
    "type": "qcow",
    "stream": {
      "type": "file",
  "file_path": "/mnt/image.qcow"
    }
}

You can use the raw format to describe a thick-provisioned, raw block device export that is rbd export –export-format 1 SNAP_SPEC. You can link the raw format data to any supported stream source:

Example

{
    "type": "raw",
    "stream": {
      "type": "file",
      "file_path": "/mnt/image-head.raw"
    },
    "snapshots": [
        {
            "type": "raw",
            "name": "snap1",
            "stream": {
              "type": "file",
       "file_path": "/mnt/image-snap1.raw"
            }
        },
    ] (optional oldest to newest ordering of snapshots)
}

The inclusion of the snapshots array is optional and currently only supports thick-provisioned raw snapshot exports.

id="streams_block"]

3.4. Streams

File stream

You can use the file stream to import from a locally accessible POSIX file source.

Syntax

{
    <format unique parameters>
    "stream": {
        "type": "file",
        "file_path": "FILE_PATH"
    }
}

For example, to import a raw-format image from a file located at /mnt/image.raw, the source-spec JSON file is:

Example

{
    "type": "raw",
    "stream": {
        "type": "file",
        "file_path": "/mnt/image.raw"
    }
}

HTTP stream

You can use the HTTP stream to import from a remote HTTP or HTTPS web server.

Syntax

{
    <format unique parameters>
    "stream": {
        "type": "http",
        "url": "URL_PATH"
    }
}

For example, to import a raw-format image from a file located at http://download.ceph.com/image.raw, the source-spec JSON file is:

Example

{
    "type": "raw",
    "stream": {
        "type": "http",
        "url": "http://download.ceph.com/image.raw"
    }
}

S3 stream

You can use the s3 stream to import from a remote S3 bucket.

Syntax

{
    <format unique parameters>
    "stream": {
        "type": "s3",
        "url": "URL_PATH",
        "access_key": "ACCESS_KEY",
        "secret_key": "SECRET_KEY"
    }
}

For example, to import a raw-format image from a file located at http://s3.ceph.com/bucket/image.raw, its source-spec JSON is encoded as follows:

Example

{
    "type": "raw",
    "stream": {
        "type": "s3",
        "url": "http://s3.ceph.com/bucket/image.raw",
        "access_key": "NX5QOQKC6BH2IDN8HC7A",
        "secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn"
    }
}

3.5. Preparing the live migration process

You can prepare the default live migration process for RBD images within the same Red Hat Ceph Storage cluster. The rbd migration prepare command accepts all the same layout options as the rbd create command. The rbd create command allows changes to the on-disk layout of the immutable image. If you only want to change the on-disk layout and want to keep the original image name, skip the migration_target argument. All clients using the source image must be stopped before preparing a live migration. The prepare step will fail if it finds any running clients with the image open in read/write mode. You can restart the clients using the new target image once the prepare step is completed.

Note

You cannot restart the clients using the source image as it will result in a failure.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Two block device pools.
  • One block device image.

Procedure

  1. Prepare the live migration within the storage cluster:

    Syntax

    rbd migration prepare SOURCE_POOL_NAME/SOURCE_IMAGE_NAME TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration prepare sourcepool1/sourceimage1 targetpool1/sourceimage1

    OR

    If you want to rename the source image:

    Syntax

    rbd migration prepare SOURCE_POOL_NAME/SOURCE_IMAGE_NAME TARGET_POOL_NAME/NEW_SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration prepare sourcepool1/sourceimage1 targetpool1/newsourceimage1

    In the example, newsourceimage1 is the renamed source image.

  2. You can check the current state of the live migration process with the following command:

    Syntax

    rbd status TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd status targetpool1/sourceimage1
    Watchers: none
    Migration:
    source: sourcepool1/sourceimage1 (adb429cb769a)
    destination: targetpool2/testimage1 (add299966c63)
    state: prepared

    Important

    During the migration process, the source image is moved into the RBD trash to prevent mistaken usage.

    Example

    [ceph: root@rbd-client /]# rbd info sourceimage1
    rbd: error opening image sourceimage1: (2) No such file or directory

    Example

    [ceph: root@rbd-client /]# rbd trash ls --all sourcepool1
    adb429cb769a sourceimage1

3.6. Preparing import-only migration

You can initiate the import-only live migration process by running the rbd migration prepare command with the --import-only and either, --source-spec or --source-spec-path options, passing a JSON document that describes how to access the source image data directly on the command line or from a file.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A bucket and an S3 object are created.

Procedure

  1. Create a JSON file:

    Example

    [ceph: root@rbd-client /]# cat testspec.json
     {
       "type": "raw",
        "stream": {
            "type": "s3",
            "url": "http:10.74.253.18:80/testbucket1/image.raw",
           "access_key": "RLJOCP6345BGB38YQXI5",
           "secret_key": "oahWRB2ote2rnLy4dojYjDrsvaBADriDDgtSfk6o"
      }

  2. Prepare the import-only live migration process:

    Syntax

    rbd migration prepare --import-only --source-spec-path "JSON_FILE" TARGET_POOL_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration prepare --import-only --source-spec-path "testspec.json" targetpool1

    Note

    The rbd migration prepare command accepts all the same image options as the rbd create command.

  3. You can check the status of the import-only live migration:

    Example

    [ceph: root@rbd-client /]# rbd status targetpool1/sourceimage1
    Watchers: none
    Migration:
    source: {"stream":{"access_key":"RLJOCP6345BGB38YQXI5","secret_key":"oahWRB2ote2rnLy4dojYjDrsvaBADriDDgtSfk6o","type":"s3","url":"http://10.74.253.18:80/testbucket1/image.raw"},"type":"raw"}
    destination: targetpool1/sourceimage1 (b13865345e66)
    state: prepared

3.7. Executing the live migration process

After you prepare for the live migration, you must copy the image blocks from the source image to the target image.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Two block device pools.
  • One block device image.

Procedure

  1. Execute the live migration:

    Syntax

    rbd migration execute TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration execute targetpool1/sourceimage1
    Image migration: 100% complete...done.

  2. You can check the feedback on the progress of the migration block deep-copy process:

    Syntax

    rbd status TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd status targetpool1/sourceimage1
    Watchers: none
    Migration:
    source: sourcepool1/testimage1 (adb429cb769a)
    destination: targetpool1/testimage1 (add299966c63)
    state: executed

3.8. Committing the live migration process

You can commit the migration, once the live migration has completed deep-copying all the data blocks from the source image to the target image.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Two block device pools.
  • One block device image.

Procedure

  1. Commit the migration, once deep-copying is completed:

    Syntax

    rbd migration commit TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration commit targetpool1/sourceimage1
    Commit image migration: 100% complete...done.

Verification

Committing the live migration will remove the cross-links between the source and target images, and also removes the source image from the source pool:

Example

[ceph: root@rbd-client /]# rbd trash list --all sourcepool1

3.9. Aborting the live migration process

You can revert the live migration process. Aborting live migration reverts the prepare and execute steps.

Note

You can abort only if you have not committed the live migration.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Two block device pools.
  • One block device image.

Procedure

  1. Abort the live migration process:

    Syntax

    rbd migration abort TARGET_POOL_NAME/SOURCE_IMAGE_NAME

    Example

    [ceph: root@rbd-client /]# rbd migration abort targetpool1/sourceimage1
    Abort image migration: 100% complete...done.

Verification

When the live migration process is aborted, the target image is deleted and access to the original source image is restored in the source pool:

Example

[ceph: root@rbd-client /]# rbd ls sourcepool1
sourceimage1