Chapter 3. Using the Block Storage backup service
You can use the Block Storage backup service to perform full or incremental backups, and to restore a backup to a volume.
3.1. Full backups
3.1.1. Creating a full volume backup
To back up a volume, use the cinder
backup-create command. By default, this command creates a full backup of the volume. If the volume has existing backups, you can choose to create an incremental backup instead. For more information, see Section 3.2.2, “Performing incremental backups”.
Before Red Hat OpenStack Platform (RHOSP) version 16, the
cinder backup-create command created incremental backups after the first full Ceph volume backup to a Ceph Storage back end. In RHOSP version 16 and later, you must use the
--incremental option to create incremental volume backups. If you do not use the
--incremental option with the
cinder backup-create command, the default setting creates full backups. For more information, see Section 3.2.2, “Performing incremental backups”.
You can create backups of volumes you have access to. This means that users with administrative privileges can back up any volume, regardless of owner. For more information, see Section 3.1.2, “Creating a volume backup as an admin”.
View the ID or Display Name of the volume you want to back up:
# cinder list
Back up the volume:
# cinder backup-create _VOLUME_
Replace VOLUME with the
Display Nameof the volume you want to back up. For example:
+-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | e9d15fc7-eeae-4ca4-aa72-d52536dc551d | | name | None | | volume_id | 5f75430a-abff-4cc7-b74e-f808234fa6c5 | +-----------+--------------------------------------+
volume_idof the resulting backup is identical to the ID of the source volume.
Verify that the volume backup creation is complete:
# cinder backup-list
The volume backup creation is complete when the
Statusof the backup entry is available.
3.1.2. Creating a volume backup as an admin
Users with administrative privileges can back up any volume managed by Red Hat OpenStack Platform. When an admin user backs up a volume that is owned by a non-admin user, the backup is hidden from the volume owner by default.
As an admin user, you can use the following command to back up a volume and make the backup available to a specific tenant:
# cinder --os-auth-url <KEYSTONEURL> --os-tenant-name <TENANTNAME> --os-username <USERNAME> --os-password <PASSWD> backup-create <VOLUME>
Replace the following variables according to your environment requirements:
- <TENANTNAME> is the name of the tenant where you want to make the backup available.
- <USERNAME> and <PASSWD> are the username and password credentials of a user within <TENANTNAME>.
- <VOLUME> is the name or ID of the volume that you want to back up.
<KEYSTONEURL> is the URL endpoint of the Identity service, which is typically http://IP:5000/v2, where IP is the IP address of the Identity service host. When you perform this operation, the size of the resulting backup counts against the quota of
TENANTNAMErather than the quota of the tenant admin.
3.1.3. Exporting the metadata of a volume backup
You can export and store the metadata of a volume backup so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss.
Run the following command:
# cinder backup-export _BACKUPID_
Replace <BACKUPID> with the ID or name of the volume backup:
+----------------+------------------------------------------+ | Property | Value | +----------------+------------------------------------------+ | backup_service | cinder.backup.drivers.swift | | backup_url | eyJzdGF0dXMiOiAiYXZhaWxhYmxlIiwgIm9iam...| | | ...4NS02ZmY4MzBhZWYwNWUiLCAic2l6ZSI6IDF9 | +----------------+------------------------------------------+
The volume backup metadata consists of the
3.1.4. Backing up an in-use volume
You can create a backup of an in-use volume with the
--force option when the Block Storage back end snapshot is supported.
To use the
--force option, the Block Storage back end snapshot must be supported. You can verify snapshot support by checking the documentation for the back end that you are using.
By using the
--force option, you acknowledge that you are not quiescing the drive before performing the backup. Using this method creates a crash-consistent, but not application-consistent, backup. This means that the backup does not have an awareness of which applications were running when the backup was performed. However, the data is intact.
To create a backup of an in-use volume, run:
# cinder backup-create _VOLUME_ --incremental --force
3.1.5. Backing up a snapshot
You can create a full backup from a snapshot by using the volume ID that is associated with the snapshot.
Locate the snapshot ID of the snapshot to backup using
cinder snapshot list.
# cinder snapshot-list --volume-id _VOLUME_ID_
If the snapshot is named, then you can use the following example to locate the
# cinder snapshot-show _SNAPSHOT_NAME_
Create the backup of a snapshot:
# cinder backup-create _VOLUME_ --snapshot-id=_SNAPSHOT_ID_Note
Snapshot-based backups of NFS volumes fail when you use the
--snapshot-idoption. This is a known issue.
3.1.6. Backing up and restoring across edge sites
You can back up and restore Block Storage service (cinder) volumes across distributed compute node (DCN) architectures in edge site and availability zones. The
cinder-backup service runs in the central availability zone (AZ), and backups are stored in the central AZ. The Block Storage service does not store backups at DCN sites.
The central site is deployed with the
cinder-backup.yamlenvironment file located in
/usr/share/openstack-tripleo-heat-templates/environments. For more information, see Block Storage backup service deployment.
- The Block Storage service (cinder) CLI is available.
All sites must use a common
openstackcephx client name. For more information, see Creating a Ceph key for external access.
Create a backup of a volume in the first DCN site:
$ cinder --os-volume-api-version 3.51 backup-create --name <volume_backup> --availability-zone <az_central> <edge_volume>
<volume_backup>with a name for the volume backup.
<az_central>with the name of the central availability zone that hosts the
<edge_volume>with the name of the volume that you want to back up.Note
If you experience issues with Ceph keyrings, you might need to restart the
cinder-backupcontainer so that the keyrings copy from the host to the container successfully.
Restore the backup to a new volume in the second DCN site:
$ cinder --os-volume-api-version 3.51 create --availability-zone <az_2> --name <new_volume> --backup-id <volume_backup> <volume_size>
<az_2>with the name of the availability zone where you want to restore the backup.
<new_volume>with a name for the new volume.
<volume_backup>with the name of the volume backup that you created in the previous step.
<volume_size>with a value in GB equal to or greater than the size of the original volume.
3.2. Incremental backups
Using the Block Storage backup service, you can perform incremental backups.
3.2.1. Performance considerations
Some backup features like incremental and data compression can impact performance. Incremental backups have a performance impact because all of the data in a volume must be read and checksummed for both the full and each incremental backup.
You can use data compression with non-Ceph backends. Enabling data compression requires additional CPU power but uses less network bandwidth and storage space overall.
The multipathing configuration also impacts performance. If you attach multiple volumes without enabling multipathing, you might not be able to connect or have full network capabilities, which impacts performance.
You can use the advanced configuration options to enable or disable compression, define the number of processes, and add additional CPU resources. For more information, see Appendix B, Advanced Block Storage backup configuration options.
22.214.171.124. Impact of backing up from a snapshot
Some back ends support creating a backup from a snapshot. A driver that supports this feature can directly attach a snapshot, which is faster than cloning the snapshot into a volume to be able to attach to it. In general, this feature can affect performance because of the extra step of creating the volume from a snapshot.
3.2.2. Performing incremental backups
By default, the
cinder backup-create command creates a full backup of a volume. However, if the volume has existing backups, you can create an incremental backup.
Incremental backups are fully supported on NFS, Object Storage (swift), and Red Hat Ceph Storage backup repositories.
An incremental backup captures any changes to the volume since the last full or incremental backup. Performing numerous, regular, full backups of a volume can become resource intensive because the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage.
To create an incremental volume backup, use the
--incrementaloption with the following command:
# cinder backup-create _VOLUME_ --incremental
Replace VOLUME with the
Display Nameof the volume you want to back up.
You cannot delete a full backup if it already has an incremental backup. If a full backup has multiple incremental backups, you can only delete the latest one.
3.3. Canceling a backup
To cancel a backup, an administrator must request a force delete on the backup.
This operation is not supported if you use the Ceph or RBD back ends.
Run the following command:
# openstack volume backup delete --force <backup>
After you complete the cancellation and the backup no longer appears in the backup listings, there can be a slight delay for the backup to be successfully canceled. To verify that the backup is successfully canceled, the backing-up status in the source resource stops.
Before Red Hat OpenStack version 12, the
backing-up status was stored in the volume, even when backing up a snapshot. Therefore, when backing up a snapshot, any delete operation on the snapshot that followed a cancellation could result in an error if the snapshot was still mapped. In Red Hat OpenStack Platform version 13 and later, ongoing restoration operations can be canceled on any of the supported backup drivers.
3.4. Viewing and modifying tenant backup quota
Normally, you can use the dashboard to modify tenant storage quotas, for example, the number of volumes, volume storage, snapshots, or other operational limits that a tenant can have. However, the functionality to modify backup quotas with the dashboard is not yet available.
You must use the command-line interface to modify backup quotas.
To view the storage quotas of a specific tenant (TENANT_ID), run the following command:
# cinder quota-show TENANT_ID
To update the maximum number of backups (MAXNUM) that can be created in a specific tenant, run the following command:
# cinder quota-update --backups MAXNUM TENANT_ID
To update the maximum total size of all backups (MAXGB) within a specific tenant, run the following command:
# cinder quota-update --backup-gigabytes MAXGB TENANT_ID
To view the storage quota usage of a specific tenant, run the following command:
# cinder quota-usage TENANT_ID
3.5. Restoring from backups
After a database failure or another type of event that results in data loss, use the backups you created to restore data.
If you configure the
cinder-backup service to use the Ceph RBD driver, you can restore backup volumes only to an RBD-based Block Storage (cinder) back end.
3.5.1. Restoring a volume from a backup
To create a new volume from a backup, complete the following steps.
Find the ID of the volume backup you want to use:
# cinder backup-list
Ensure that the Volume ID matches the ID of the volume that you want to restore.
Restore the volume backup:
# cinder backup-restore _BACKUP_ID_
Replace BACKUP_ID with the ID of the volume backup you want to use.
If you no longer need the backup, delete it:
# cinder backup-delete _BACKUP_ID_
If you need to restore a backed up volume to a volume of a particular type, use the
--volumeoption to restore a backup to a specific volume:
# cinder backup-restore _BACKUP_ID --volume VOLUME_ID_Note
If you restore a volume from an encrypted backup, then the destination volume type must also be encrypted.
3.5.2. Restoring a volume after a Block Storage database loss
When a Block Storage database loss occurs, you cannot restore a volume backup because the database contains metadata that the volume backup service requires. However, after you create the volume backup, you can export and store the metadata, which consists of
backup_url values, so that when a database loss occurs, you can restore the volume backup. For more information see Section 3.1.1, “Creating a full volume backup”).
If you exported and stored this metadata, then you can import it to a new Block Storage database, which allows you to restore the volume backup.
For incremental backups, you must import all exported data before you can restore one of the incremental backups.
As a user with administrative privileges, run the following command:
# cinder backup-import _backup_service_ _backup_url_
backup_urlwith the metadata you exported. For example, using the exported metadata from Section 3.1.1, “Creating a full volume backup”:
# cinder backup-import cinder.backup.drivers.swift eyJzdGF0dXMi...c2l6ZSI6IDF9 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | id | 77951e2f-4aff-4365-8c64-f833802eaa43 | | name | None | +----------+--------------------------------------+
- After you import the metadata into the Block Storage service database, you can restore the volume as normal, see Section 3.5.1, “Restoring a volume from a backup”.
3.5.3. Canceling a backup restore
To cancel a backup restore operation, alter the status of the backup to anything other than
restoring. You can use the
error state to minimize confusion regarding whether the restore was successful or not. Alternatively, you can change the value to
$ openstack volume backup set --state error BACKUP_ID
Backup cancellation is an asynchronous action, because the backup driver must detect the status change before it cancels the backup. When the status changes to
available in the destination volume, the cancellation is complete.
This feature is not currently available on RBD backups.
If a restore operation is canceled after it starts, the destination volume is useless, because there is no way of knowing how much data, if any, was actually restored.
There are two common scenarios that cause many of the issues that occur with the backup service:
cinder-backupservice starts, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause services to fail.
- When backups are requested, the backup service connects to the volume service and attaches the requested volume. Problems with this connection are evident only during backup time.
In either case, the logs contain messages that describe the error.
For more information about log files and services, see Log Files for OpenStack Services in the Logging, Monitoring and Troubleshooting Guide.
For more general information about log locations, and troubleshooting suggestions, see Block Storage (cinder) Log Files in the Logging, Monitoring and Troubleshooting Guide.
3.6.1. Verifying services
You can diagnose many issues by verifying that services are available and by checking log files for error messages. For more information about the key services and their interactions, see Section 1.2, “How backups and restores work”.
After you verify the status of the services, check the
cinder-backup.log file. The Block Storage Backup service log is located in
cinder showcommand on the volume to see if it is stored by the host:
# cinder show
cinder service-listcommand to view running services:
# cinder service-list +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-scheduler | hostgroup | nova | enabled | up | 2017-05-15T02:42:25.000000 | - | | cinder-volume | hostgroup@sas-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | | cinder-volume | hostgroup@ssd-pool | nova | enabled | down | 2017-05-14T03:04:01.000000 | - | +------------------+--------------------+------+---------+-------+----------------------------+-----------------+
- Verify that the expected services are available.
3.6.2. Troubleshooting tips
Backups are asynchronous. The Block Storage backup service performs a small number of static checks upon receiving an API request, such as checking for an invalid volume reference (
missing) or a volume that is
in-use or attached to an instance. The
in-use case requires you to use the
--force option means that I/O is not be quiesced and the resulting volume image may be corrupt.
If the API accepts the request, the backup occurs in the background. Usually, the CLI returns immediately even if the backup fails or is approaching failure. You can query the status of a backup by using the cinder backup API. If an error occurs, review the logs to discover the cause.
By default, Pacemaker deploys the Block Storage backup service. Pacemaker configures virtual IP addresses, containers, services, and other features as resources in a cluster to ensure that the defined set of OpenStack cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most services are through HAProxy
For information about how to use Pacemaker for troubleshooting, see Managing high availability services with Pacemaker in the High Availability Deployment and Usage guide.