16.5. Backing Up and Restoring Virtual Machines Using the Incremental Backup and Restore API

Red Hat Virtualization provides an Incremental Backup API that you can use for full backups of QCOW2 or RAW virtual disks, or incremental backups of QCOW 2 virtual disks, without any temporary snapshots. Data is backed-up in RAW format, whether the virtual disk being backed up is QCOW2 or RAW. You can restore RAW guest data and either RAW or QCOW2 disks. The Incremental Backup API is part of the RHV REST API. You can backup virtual machines that are running or that are not.

As a developer, you can use the API to develop a backup application.

Features

Backups are simpler, faster and more robust than when using the Backup and Restore API. The Incremental Backup API provides improved integration with backup applications, with new support for backing up and restoring RAW guest data, regardless of the underlying disk format.

Limitations:

  • Only disks in QCOW2 format can be backed up incrementally, not RAW format disks. The backup process saves the backed up data in RAW format.
  • Only backed up data in RAW format can be restored.
  • Incremental restore does not support restoring snapshots as they existed at the time of the backup, rather incremental restore restores only the data and not the structure of volumes or images in snapshots as they existed at the time of the backup. This limit is common in backup solutions for other systems.
  • As is commonly the case with backup solutions, incremental restore restores only the data and not the structure of volumes or images in snapshots as they existed at the time of the backup.
  • An unclean shutdown of a virtual machine, whatever the cause, might invalidate bitmaps on the disk, which invalidates the entire backup chain. Restoring an incremental backup using an invalid bitmap leads to corrupt virtual machine data.

    There is no way to detect an invalid bitmap, other than starting a backup. If the disk includes any invalid bitmaps, the operation fails.

    To recover from invalid bitmaps, you must remove the entire backup chain and run a full backup.

Table 16.1. How enabling incremental backup affects disk format:

Storage typeProvisioning typeWhen incremental backup is…​Virtual disk format is…​ 

block

thin

enabled

qcow2

block

thin

disabled

qcow2

file

thin

enabled

qcow2

file

thin

disabled

raw (sparse)

block

preallocated

enabled

qcow2 (preallocated)

block

preallocated

disabled

raw (preallocated)

file

preallocated

enabled

qcow2 (preallocated)

file

preallocated

disabled

raw (preallocated)

network

Not applicable

disabled

raw

lun

Not applicable

disabled

raw

16.5.1. The Incremental Backup Flow

A backup application that uses the Incremental Backup API must follow this sequence to back up virtual machine disks that have already been enabled for incremental backup:

  1. The backup application uses the REST API to find virtual machine disks that should be included in the backup. Only disks in QCOW2 format are included.
  2. The backup application starts a full backup or an incremental backup. The API call specifies a virtual machine ID, an optional previous checkpoint ID, and a list of disks to back up. If the API call does not specify a previous checkpoint ID, a full backup begins, which includes all data in the specified disks, based on the current state of each disk.
  3. The engine prepares the virtual machine for backup. The virtual machine can continue running during the backup.
  4. The backup application polls the engine for the backup status, until the engine reports that the backup is ready to begin.
  5. When the backup is ready to begin, the backup application creates an image transfer object for every disk included in the backup.
  6. The backup application gets a list of changed blocks from ovirt-imageio for every image transfer. If a change list is not available, the backup application gets an error.
  7. The backup application downloads changed blocks in RAW format from ovirt-imageio and stores them in the backup media. If a list of changed blocks is not available, the backup application can fall back to copying the entire disk.
  8. The backup application finalizes all image transfers.
  9. The backup application finalizes the backup using the REST API.

16.5.2. The Incremental Restore Flow

A backup application that uses the Incremental Backup API must follow this sequence to restore virtual machine disks that have been backed up:

  1. The user selects a restore point based on available backups using the backup application.
  2. The backup application creates a new disk or a snapshot with an existing disk to hold the restored data.
  3. The backup application starts an upload image transfer for every disk, specifying format is raw. This enables format conversion when uploading RAW data to a QCOW2 disk.
  4. The backup application transfers the data included in this restore point to imageio using the API.
  5. The backup application finalizes the image transfers.

16.5.3. Incremental Backup and Restore API Tasks

The Incremental Backup and Restore API is documented in the Red Hat Virtualization REST API Guide. The backup and restore flow requires the following actions.

16.5.3.1. Enabling Incremental Backup on a new virtual disk

Enable incremental backup for a virtual disk to mark it as included in an incremental backup. When adding a disk, you can enable incremental backup for every disk, either with the REST API or using the Administration Portal. You can back up existing disks that are not enabled for incremental backup using full backup or in the same way you did previously.

Note

The Manager does not require the disk to be enabled for it to be included in an incremental backup, but you can enable it to keep track of which disks are enabled.

Because incremental backup requires disks to be formatted in QCOW2, use QCOW2 format instead of RAW format.

Procedure

  1. Add a new virtual disk. For more information see Creating a Virtual Disk.
  2. When configuring the disk, select the Enable Incremental Backup checkbox.

16.5.3.2. Enabling Incremental Backup on an existing RAW virtual disk

Because incremental backup is not supported for disks in RAW format, a QCOW2 format layer must exist on top of any RAW format disks in order to use incremental backup. Creating a snapshot generates a QCOW2 layer, enabling incremental backup on all disks that are included in the snapshot, from the point at which the snapshot is created.

Warning

If the base layer of a disk uses RAW format, deleting the last snapshot and merging the top QCOW2 layer into the base layer converts the disk to RAW format, thereby disabling incremental backup if it was set. To re-enable incremental backup, you can create a new snapshot, including this disk.

Procedure

  1. In the Administration Portal, click ComputeVirtual Machines.
  2. Select a virtual machine and click the Disks tab.
  3. Click the Edit button. The Edit Disk dialog box opens.
  4. Select the Enable Incremental Backup checkbox.

16.5.3.3. Enabling incremental backup

You can use a REST API request to enable incremental backup for a virtual machine’s disk.

Procedure

  • Enable incremental backup for a new disk. For example, for a new disk on a virtual machine with id 123, send this request:

    POST /ovirt-engine/api/vms/123/diskattachments

    The request body should include backup set to incremental as part of a disk object, like this:

    <disk_attachment>
        …​
        <disk>
           …​
           <backup>incremental</backup>
           …​
        </disk>
    </disk_attachment>

The response is:

<disk_attachment>
    …​
    <disk href="/ovirt-engine/api/disks/456" id="456"/>
    …​
</disk_attachment>

Additional resources

16.5.3.4. Finding disks that are enabled for incremental backup

For the specified virtual machine, you can list the disks that are enabled for incremental backup, filtered according to the backup property.

Procedure

  1. List the disks that are attached to the virtual machine. For example, for a virtual machine with the id 123, send this request:

    GET /ovirt-engine/api/vms/123/diskattachments

    The response includes all disk_attachment objects, each of which includes one or more disk objects. For example:

    <disk_attachments>
        <disk_attachment>
           …​
           <disk href="/ovirt-engine/api/disks/456" id="456"/>
           …​
        </disk_attachment>
        …​
    </disk_attachments>
  2. Use the disk service to see the properties of a disk from the previous step. For example, for the disk with the id 456, send this request:

    GET /ovirt-engine/api/disks/456

    The response includes all properties for the disk. backup is set to none or incremental. For example:

    <disk href="/ovirt-engine/api/disks/456" id="456">
        …​
        <backup>incremental</backup>
        …​
    </disk>

16.5.3.5. Starting a full backup

After a full backup you can use the resulting checkpoint id as the start point in the next incremental backup.

When taking a backup of a running virtual machine, the process creates a scratch disk on the same storage domain as the disk being backed up. The backup process creates this disk to enable new data to be written to the running virtual machine during the backup. You can see this scratch disk in the Administration Portal during the backup. It is automatically deleted when the backup finishes.

Starting a full backup requires a request call with a body, and includes a response.

Procedure

  1. Send a request specifying a virtual machine to back up. For example, specify a virtual machine with id 123 like this:

    POST /ovirt-engine/api/vms/123/backups
  2. In the request body, specify a disk to back up. For example, to start a full backup of a disk with id 456, send the following request body:

    <backup>
        <disks>
           <disk id="456" />
           …​
        </disks>
    </backup>

    The response body should look similar to this:

    <backup id="789">
        <disks>
           <disk id="456" />
           …​
           …​
        </disks>
        <status>initializing</status>
        <creation_date>
    </backup>

    The response includes the following:

    • The backup id
    • The status of the backup, indicating that the backup is initializing.
  3. Poll the backup until the status is ready. The response includes to_checkpoint_id. Note this id and use it for from_checkpoint_id in the next incremental backup.

Additional resources

16.5.3.6. Starting an incremental backup

Once a full backup is completed for a given virtual disk, subsequent incremental backups that disk contain only the changes since the last backup. Use the value of to_checkpoint_id from the most recent backup as the value for from_checkpoint_id in the request body.

When taking a backup of a running virtual machine, the process creates a scratch disk on the same storage domain as the disk being backed up. The backup process creates this disk to enable new data to be written to the running virtual machine during the backup. You can see this scratch disk in the Administration Portal during the backup. It is automatically deleted when the backup finishes.

Starting an incremental backup or mixed backup requires a request call with a body, and includes a response.

Procedure

  1. Send a request specifying a virtual machine to back up. For example, specify a virtual machine with id 123 like this:

    POST /ovirt-engine/api/vms/123/backups
  2. In the request body, specify a disk to back up. For example, to start an incremental backup of a disk with id 456, send the following request body:

    <backup>
        <from_checkpoint_id>previous-checkpoint-uuid</from_checkpoint_id>
        <disks>
           <disk id="456" />
           …​
        </disks>
    </backup>
    Note

    In the request body, if you include a disk that is not included in the previous checkpoint, the request also runs a full backup of this disk. For example, a disk with id 789 has not been backed up yet. To add a full backup of 789 to the above request body, send a request body like this:

    <backup>
        <from_checkpoint_id>previous-checkpoint-uuid</from_checkpoint_id>
        <disks>
           <disk id="456" />
           <disk id="789" />
           …​
        </disks>
    </backup>

    The response body should look similar to this:

    <backup id="101112">
    <from_checkpoint_id>previous-checkpoint-uuid</from_checkpoint_id>
    <to_checkpoint_id>new-checkpoint-uuid</to_checkpoint_id>
        <disks>
           <disk id="456" />
           <disk id="789" />
           …​
           …​
        </disks>
        <status>initializing</status>
        <creation_date>
    </backup>

    The response includes the following:

    • The backup id.
    • The id of any disk that was included in the backup.
    • The status, indicating that the backup is initializing.
  3. Poll the backup until the status is ready. The response includes to_checkpoint_id. Note this id and use it for from_checkpoint_id in the next incremental backup.

Additional resources

16.5.3.7. Getting information about a backup

You can get information about a backup that you can use to start a new incremental backup.

The list method of the VmBackups service returns the following information about a backup:

  • the id of each disk that was backed up
  • the ids of the start and end checkpoints of the backup
  • the id of the disk image of the backup, for each disk included in the backup
  • the status of the backup
  • the date the backup was created

When the value of <status> is ready, the response includes <to_checkpoint_id> which should be used as the <from_checkpoint_id> in the next incremental backup and you can start downloading the disks to back up the virtual machine storage.

Procedure

  • To get information about a backup with id 456 of a virtual machine with id 123, send a request like this:

    GET /ovirt-engine/api/vms/456/backups/123

    The response includes the backup with id 456, with <from_checkpoint_id> 999 and <to_checkpoint_id> 666. The disks included in the backup are referenced in the <link> element.

    <backup id="456">
        <from_checkpoint_id>999</from_checkpoint_id>
        <to_checkpoint_id>666</to_checkpoint_id>
        <link href="/ovirt-engine/api/vms/456/backups/123/disks" rel="disks"/>
        <status>ready</status>
        <creation_date>
    </backup>

16.5.3.8. Getting information about the disks in a backup

You can get information about the disks that are part of the backup, including the backup mode that was used for each disk in a backup, which helps determine the mode that you use to download the backup.

The list method of the VmBackupDisks service returns the following information about a backup:

  • The id and name of each disk that was backed up.
  • The id of the disk image of the backup, for each disk included in the backup.
  • The disk format.
  • The backup behavior supported by the disk.
  • The backup type that was taken for the disk (full/incremental).

Procedure

  • To get information about a backup with id 456 of a virtual machine with id 123, send a request like this:

    GET /ovirt-engine/api/vms/456/backups/123/disks

    The response includes the disk with id 789, and the id of the disk image is 555.

    <disks>
        <disk id="789">
            <name>vm1_Disk1</name>
            <actual_size>671744</actual_size>
            <backup>incremental</backup>
            <backup_mode>full</backup_mode>
            <format>cow</format>
            <image_id>555</image_id>
            <qcow_version>qcow2_v3</qcow_version>
            <status>locked</status>
            <storage_type>image</storage_type>
            <total_size>0</total_size>
        </disk>
    </disks>

16.5.3.9. Finalizing a backup

Finalizing a backup ends the backup, unlocks resources, and performs cleanups. Use the finalize backup service method

Procedure

  • To finalize a backup of a disk with id 456 on a virtual machine with id 123, send a request like this:

    POST /vms/123/backups/456/finalize

Additional resources

16.5.3.10. Creating an image transfer object for incremental backup

When the backup is ready to download, the backup application should create an imagetransfer object, which initiates a transfer for an incremental backup.

Creating an image transfer object requires a request call with a body.

Procedure

  1. Send a request like this:

    POST /ovirt-engine/api/imagetransfers
  2. In the request body, specify the following parameters:

    • disk id
    • backup id
    • direction of the disk set to download
    • format of the disk set to raw

    For example, to transfer a backup of a disk where the id of the disk is 123 and the id of the backup is 456, send the following request body:

    <image_transfer>
        <disk id="123"/>
        <backup id="456"/>
        <direction>download</direction>
        <format>raw</format>
    </image_transfer>

Additional resources

16.5.3.11. Creating an image transfer object for incremental restore

To enable restoring raw data backed up using the incremental backup API to a QCOW2-formatted disk, the backup application should create an imagetransfer object.

When the transfer format is raw and the underlying disk format is QCOW2, uploaded data is converted on the fly to QCOW2 format when writing to storage. Uploading data from a QCOW2 disk to a RAW disk is not supported.

Creating an image transfer object requires a request call with a body.

Procedure

  1. Send a request like this:

    POST /ovirt-engine/api/imagetransfers
  2. In the request body, specify the following parameters:

    • disk id or snapshot id
    • direction of the disk set to upload
    • format of the disk set to raw

    For example, to transfer a backup of a disk where the id of the disk is 123, send the following request body:

    <image_transfer>
        <disk id="123"/>
        <direction>upload</direction>
        <format>raw</format>
    </image_transfer>

Additional resources

16.5.3.12. Listing checkpoints for a virtual machine

You can list all checkpoints for a virtual machine, including information for each checkpoint, by sending a request call.

Procedure

  • Send a request specifying a virtual machine. For example, specify a virtual machine with id 123 like this:

    GET /vms/123/checkpoints/

The response includes all the virtual machine checkpoints. Each checkpoint contains the following information:

  • The checkpoint’s disks
  • The id of the parent checkpoint
  • Creation date of the checkpoint
  • The virtual machine to which it belongs

For example:

<parent_id>, <creation_date> and the virtual machine it belongs to <vm>:
<checkpoints>
    <checkpoint id="456">
         <link href="/ovirt-engine/api/vms/vm-uuid/checkpoints/456/disks" rel="disks"/>
         <parent_id>parent-checkpoint-uuid</parent_id>
         <creation_date>xxx</creation_date>
         <vm href="/ovirt-engine/api/vms/123" id="123"/>
    </checkpoint>
</checkpoints>

Additional resources

16.5.3.13. Listing a specific checkpoint for a virtual machine

You can list information for a specific checkpoint for a virtual machine by sending a request call.

Procedure

  • Send a request specifying a virtual machine. For example, specify a virtual machine with id 123 and checkpoint id 456 like this:

    GET /vms/123/checkpoints/456

The response includes the following information for the checkpoint:

  • The checkpoint’s disks
  • The id of the parent checkpoint
  • Creation date of the checkpoint
  • The virtual machine to which it belongs

For example:

<checkpoint id="456">
     <link href="/ovirt-engine/api/vms/vm-uuid/checkpoints/456/disks" rel="disks"/>
     <parent_id>parent-checkpoint-uuid</parent_id>
     <creation_date>xxx</creation_date>
     <vm href="/ovirt-engine/api/vms/123" id="123"/>
</checkpoint>

Additional resources

16.5.3.14. Removing a checkpoint

You can remove a checkpoint of a virtual machine by sending a DELETE request. You can remove a checkpoint on a virtual machine whether it is running or not.

Procedure

  • Send a request specifying a virtual machine and a checkpoint. For example, specify a virtual machine with id 123 and a checkpoint with id 456 like this:

    DELETE /vms/123/checkpoints/456/

Additional resources

16.5.3.15. Using the imageio API to transfer backup data

Image transfer APIs start and stop an image transfer. The result is a transfer URL.

You use the imageio API to actually transfer the data from the transfer URL.

For complete information on using the imageio API, see the ovirt-imageio Images API reference.

Table 16.2. imageio Image API methods used in incremental backup and restore

API requestDescriptionimageio Image API reference section

OPTIONS /images/{ticket-id} HTTP/1.1

Gets the server options, to find out what features the server supports.

See OPTIONS

GET /images/{ticket-id}/extents

Gets information about disk image content and allocation, or about blocks that changed during an incremental backup. This information is known as extent information.

See EXTENTS

GET /images/{ticket-id}/extent?context=dirty

The program doing image transfer needs to download changes from the backup. These changes are know as dirty extents. To download changes, send a request like this:

See EXTENTSExamplesRequest dirty extents

PUT /images/{ticket-id}

The backup application creates a new disk or a snapshot with an existing disk to hold the restored data.

See PUT

Additional resources

The Red Hat Virtualization Python SDK includes several implementation examples you can use to get started with transferring backups: