Chapter 3. Using the Block Storage backup service

You can use the Block Storage backup service to perform full or incremental backups, and to restore a backup to a volume.

3.1. Full backups

3.1.1. Creating a full volume backup

To back up a volume, use the cinder backup-create command. By default, this command creates a full backup of the volume. If the volume has existing backups, you can choose to create an incremental backup instead. For more information, see Section 3.2.2, “Performing incremental backups”.

Note

Prior to Red Hat OpenStack Platform version 16, the cinder backup-create command created incremental backups after the first full Ceph volume backup to a Ceph Storage back end. In RHOSP version 16 and later, you must use the --incremental option to create incremental volume backups. If the --incremental option is not used with the cinder backup-create command, the default setting creates full backups. For more information, see Section 3.2.2, “Performing incremental backups”.

You can create backups of volumes you have access to. This means that users with administrative privileges can back up any volume, regardless of owner. For more information, see Section 3.1.2, “Creating a volume backup as an admin”.

Procedure

  1. View the ID or Display Name of the volume you want to back up: 

    # cinder list

  2. Back up the volume: 

    # cinder backup-create _VOLUME_

    Replace VOLUME with the ID or Display Name of the volume you want to back up. For example: 

      +-----------+--------------------------------------+
  |  Property |                Value                 |
  +-----------+--------------------------------------+
  |     id    | e9d15fc7-eeae-4ca4-aa72-d52536dc551d |
  |    name   |                 None                 |
  | volume_id | 5f75430a-abff-4cc7-b74e-f808234fa6c5 |
  +-----------+--------------------------------------+

    The volume_id of the resulting backup is identical to the ID of the source volume.

  3. Verify that the volume backup creation is complete: 

    # cinder backup-list
  4. The volume backup creation is complete when the Status of the backup entry is available.

3.1.2. Creating a volume backup as an admin

Users with administrative privileges can back up any volume managed by Red Hat OpenStack Platform. When an admin user backs up a volume that is owned by a non-admin user, the backup is hidden from the volume owner by default.

Procedure

  • As an admin user, you can use the following command to back up a volume and make the backup available to a specific project: 

    # cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <PROJECTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>

    Replace the following variables according to your environment requirements:

  • <PROJECTNAME> is the name of the project (tenant) where you want to make the backup available.
  • <USERNAME> and <PASSWD> are the username and password credentials of a user within <PROJECTNAME>.
  • <VOLUME> is the name or ID of the volume that you want to back up.
  • <KEYSTONEURL> is the URL endpoint of the Identity service, which is typically http://<IP>:5000/v2 where <IP> is the IP address of the Identity service host. When you perform this operation, the size of the resulting backup counts against the quota of <PROJECTNAME> rather than the quota of the project admin.

3.1.3. Exporting the metadata of a volume backup

You can export and store the metadata of a volume backup so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss.

Procedure

  • Run the following command: 

    # cinder backup-export _BACKUPID_

    Replace <BACKUPID> with the ID or name of the volume backup: 

    +----------------+------------------------------------------+
|    Property    |                Value                     |
+----------------+------------------------------------------+
| backup_service |     cinder.backup.drivers.swift          |
|   backup_url   | eyJzdGF0dXMiOiAiYXZhaWxhYmxlIiwgIm9iam...|
|                | ...4NS02ZmY4MzBhZWYwNWUiLCAic2l6ZSI6IDF9 |
+----------------+------------------------------------------+

The volume backup metadata consists of the backup_service and backup_url values.

3.1.4. Backing up an in-use volume

You can create a backup of an in-use volume with the --force option when the Block Storage back end snapshot is supported.

Note

To use the --force option, the Block Storage back end snapshot must be supported. You can verify snapshot support by checking the documentation for the back end that you are using.

By using the --force option, you acknowledge that you are not quiescing the drive before performing the backup. Using this method creates a crash-consistent, but not application-consistent, backup. This means that the backup does not have an awareness of which applications were running when the backup was performed. However, the data is intact.

Procedure

  • To create a backup of an in-use volume, run: 

    # cinder backup-create _VOLUME_ --incremental --force

3.1.5. Backing up a snapshot

You can create a full backup from a snapshot using the volume ID that is associated with the snapshot.

Procedure

  1. Locate the snapshot ID of the snapshot to backup using cinder snapshot list. 

    # cinder snapshot-list --volume-id _VOLUME_ID_

  2. If the snapshot is named, then you can use the following example to locate the ID: 

    # cinder snapshot-show _SNAPSHOT_NAME_

  3. Create the backup of a snapshot: 

    # cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_

3.2. Incremental backups

Using the Block Storage backup service, you can perform incremental backups.

3.2.1. Performance considerations

Some backup features like incremental and data compression can impact performance. Incremental backups have a performance impact because all of the data in a volume must be read and checksummed for both the full and each incremental backup.

You can use data compression with non-Ceph backends. Enabling data compression requires additional CPU power but uses less network bandwidth and storage space overall.

The multipathing configuration also impacts performance. If you attach multiple volumes without enabling multipathing, you might not be able to connect or have full network capabilities, which impacts performance.

You can use the advanced configuration options to enable or disable compression, define the number of processes, and add additional CPU resources. For more information, see Appendix B, Advanced Block Storage backup configuration options.

3.2.1.1. Impact of backing up from a snapshot

Some back ends support creating a backup from a snapshot. A driver that supports this feature can directly attach a snapshot, which is faster than cloning the snapshot into a volume to be able to attach to it. In general, this feature can affect performance because of the extra step of creating the volume from a snapshot.

3.2.2. Performing incremental backups

By default, the cinder backup-create command creates a full backup of a volume. However, if the volume has existing backups, you can create an incremental backup.

Incremental backups are fully supported on NFS, Object Storage (swift), and Red Hat Ceph Storage backup repositories.

An incremental backup captures any changes to the volume since the last full or incremental backup. Performing numerous, regular, full backups of a volume can become resource intensive because the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage.

Procedure

  • To create an incremental volume backup, use the --incremental option with the following command: 

    # cinder backup-create _VOLUME_ --incremental

    Replace VOLUME with the ID or Display Name of the volume you want to back up.

Note

You cannot delete a full backup if it already has an incremental backup. If a full backup has multiple incremental backups, you can only delete the latest one.

3.3. Canceling a backup

To cancel a backup, an administrator must request a force delete on the backup.

Important

This operation is not supported if you use the Ceph or RBD back ends.

Procedure

  • Run the following command: 

    # openstack volume backup delete --force <backup>

After you complete the cancellation and the backup no longer appears in the backup listings, there can be a slight delay for the backup to be successfully canceled. To verify that the backup is successfully canceled, the backing-up status in the source resource stops.

Note

Before Red Hat OpenStack version 12, the backing-up status was stored in the volume, even when backing up a snapshot. Therefore, when backing up a snapshot, any delete operation on the snapshot that followed a cancellation could result in an error if the snapshot was still mapped. In Red Hat OpenStack Platform version 13 and later, ongoing restoration operations can be canceled on any of the supported backup drivers.

3.4. Viewing and modifying project backup quota

Normally, you can use the dashboard to modify project storage quotas, for example, the number of volumes, volume storage, snapshots, or other operational limits that a project can have. However, the functionality to modify backup quotas with the dashboard is not yet available.

You must use the command-line interface to modify backup quotas.

Procedure

  1. To view the storage quotas of a specific project (tenant), enter the following command: 

    # cinder quota-show <PROJECT_ID>

  2. Update the maximum number of backups, <MAXNUM>, that can be created in a specific project: 

    # cinder quota-update --backups <MAXNUM> <PROJECT_ID>

  3. Update the maximum total size of all backups, <MAXGB>, within a specific project: 

    # cinder quota-update --backup-gigabytes MAXGB <PROJECT_ID>

  4. View the storage quota usage of a specific project: 

    # cinder quota-usage <PROJECT_ID>

3.5. Restoring from backups

3.5.1. Restoring a volume from a backup

To create a new volume from a backup, complete the following steps.

Procedure

  1. Find the ID of the volume backup you want to use: 

    # cinder backup-list

    Ensure that the Volume ID matches the ID of the volume that you want to restore.

  2. Restore the volume backup: 

    # cinder backup-restore _BACKUP_ID_

    Replace BACKUP_ID with the ID of the volume backup you want to use.

  3. If you no longer need the backup, delete it: 

    # cinder backup-delete _BACKUP_ID_

  4. If you need to restore a backed up volume to a volume of a particular type, use the --volume option to restore a backup to a specific volume: 

    # cinder backup-restore _BACKUP_ID --volume VOLUME_ID_
    Note

    If you restore a volume from an encrypted backup, then the destination volume type must also be encrypted.

3.5.2. Restoring a volume after a Block Storage database loss

When a Block Storage database loss occurs, you cannot restore a volume backup because the database contains metadata that the volume backup service requires. However, after you create the volume backup, you can export and store the metadata, which consists of backup_service and backup_url values, so that when a database loss occurs, you can restore the volume backup. For more information see Section 3.1.1, “Creating a full volume backup”).

If you exported and stored this metadata, then you can import it to a new Block Storage database, which allows you to restore the volume backup.

Note

For incremental backups, you must import all exported data before you can restore one of the incremental backups.

Procedure

  1. As a user with administrative privileges, run the following command: 

    # cinder backup-import _backup_service_ _backup_url_

    Replace backup_service and backup_url with the metadata you exported. For example, using the exported metadata from Section 3.1.1, “Creating a full volume backup”: 

    # cinder backup-import cinder.backup.drivers.swift eyJzdGF0dXMi...c2l6ZSI6IDF9
+----------+--------------------------------------+
| Property |                Value                 |
+----------+--------------------------------------+
|    id    | 77951e2f-4aff-4365-8c64-f833802eaa43 |
|   name   |                 None                 |
+----------+--------------------------------------+
  2. After you import the metadata into the Block Storage service database, you can restore the volume as normal, see Section 3.5.1, “Restoring a volume from a backup”.

3.5.3. Canceling a backup restore

To cancel a backup restore operation, alter the status of the backup to anything other than restoring. You can use the error state to minimize confusion regarding whether the restore was successful or not. Alternatively, you can change the value to available. 

$ openstack volume backup set --state error BACKUP_ID
Note

Backup cancellation is an asynchronous action, because the backup driver must detect the status change before it cancels the backup. When the status changes to available in the destination volume, the cancellation is complete.

Note

This feature is not currently available on RBD backups.

Warning

If a restore operation is canceled after it starts, the destination volume is useless, because there is no way of knowing how much data, if any, was actually restored.

3.6. Troubleshooting

There are two common scenarios that cause many of the issues that occur with the backup service:

  • When the cinder-backup service starts, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause services to fail.
  • When backups are requested, the backup service connects to the volume service and attaches the requested volume. Problems with this connection are evident only during backup time.

In either case, the logs contain messages that describe the error.

For more information about log files and services, see Log Files for OpenStack Services in the Logging, Monitoring and Troubleshooting Guide.

For more general information about log locations, and troubleshooting suggestions, see Block Storage (cinder) Log Files in the Logging, Monitoring and Troubleshooting Guide.

3.6.1. Verifying services

You can diagnose many issues by verifying that services are available and by checking log files for error messages. For more information about the key services and their interactions, see Section 1.2, “How backups and restores work”.

After you verify the status of the services, check the cinder-backup.log file. The Block Storage Backup service log is located in /var/log/containers/cinder]/cinder-backup.log.

Procedure

  1. Run the cinder show command on the volume to see if it is stored by the host: 

    # cinder show

  2. Run the cinder service-list command to view running services: 

    # cinder service-list
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host               | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | hostgroup          | nova | enabled | up    | 2017-05-15T02:42:25.000000 | -               |
| cinder-scheduler | hostgroup          | nova | enabled | up    | 2017-05-15T02:42:25.000000 | -               |
| cinder-volume    | hostgroup@sas-pool | nova | enabled | down  | 2017-05-14T03:04:01.000000 | -               |
| cinder-volume    | hostgroup@ssd-pool | nova | enabled | down  | 2017-05-14T03:04:01.000000 | -               |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
  3. Verify that the expected services are available.

3.6.2. Troubleshooting tips

Backups are asynchronous. The Block Storage backup service performs a small number of static checks upon receiving an API request, such as checking for an invalid volume reference (missing) or a volume that is in-use or attached to an instance. The in-use case requires you to use the --force option.

Note

Using the --force option means that I/O is not be quiesced and the resulting volume image may be corrupt.

If the API accepts the request, the backup occurs in the background. Usually, the CLI returns immediately even if the backup fails or is approaching failure. You can query the status of a backup by using the cinder backup API. If an error occurs, review the logs to discover the cause.

3.6.3. Pacemaker

By default, Pacemaker deploys the Block Storage backup service. Pacemaker configures virtual IP addresses, containers, services, and other features as resources in a cluster to ensure that the defined set of OpenStack cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most services are through HAProxy

For information about how to use Pacemaker for troubleshooting, see Managing high availability services with Pacemaker in the High Availability Deployment and Usage guide.