Chapter 4. Troubleshooting the Block Storage backup service

There are two common scenarios that cause many of the issues that occur with the backup service:

  • When the cinder-backup service starts, it connects to its configured backend and uses this as a target for backups. Problems with this connection can cause services to fail.
  • When backups are requested, the backup service connects to the volume service and attaches the requested volume. Problems with this connection are evident only during backup time.

In either case, the logs contain messages that describe the error.

For more information about log files and services, see Location of log files for OpenStack services in the Logging, Monitoring and Troubleshooting Guide.

For more information about log locations and troubleshooting suggestions, see Block Storage (cinder) Log Files in the Logging, Monitoring and Troubleshooting Guide.

4.1. Verifying services

You can diagnose many issues by verifying that services are available and by checking log files for error messages.

After you verify the status of the services, check the cinder-backup.log file. The Block Storage Backup service log is located in /var/log/containers/cinder]/cinder-backup.log.

Procedure

  1. Run the cinder show command on the volume to see if it is stored by the host: 

    # cinder show

  2. Run the cinder service-list command to view running services: 

    # cinder service-list
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host               | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | hostgroup          | nova | enabled | up    | 2017-05-15T02:42:25.000000 | -               |
| cinder-scheduler | hostgroup          | nova | enabled | up    | 2017-05-15T02:42:25.000000 | -               |
| cinder-volume    | hostgroup@sas-pool | nova | enabled | down  | 2017-05-14T03:04:01.000000 | -               |
| cinder-volume    | hostgroup@ssd-pool | nova | enabled | down  | 2017-05-14T03:04:01.000000 | -               |
+------------------+--------------------+------+---------+-------+----------------------------+-----------------+
  3. Verify that the expected services are available.

4.2. Querying the status of a failed backup

Backups are asynchronous. The Block Storage backup service performs a small number of static checks upon receiving an API request, such as checking for an invalid volume reference (missing) or a volume that is in-use or attached to an instance. The in-use case requires you to use the --force option.

Note

Using the --force option means that I/O is not be quiesced and the resulting volume image may be corrupt.

If the API accepts the request, the backup occurs in the background. Usually, the CLI returns immediately even if the backup fails or is approaching failure. You can query the status of a backup by using the cinder backup API. If an error occurs, review the logs to discover the cause.

4.3. Using Pacemaker to manage resources

By default, Pacemaker deploys the Block Storage backup service. Pacemaker configures virtual IP addresses, containers, services, and other features as resources in a cluster to ensure that the defined set of Red Hat OpenStack Platform cluster resources are running and available. When a service or an entire node in a cluster fails, Pacemaker can restart the resource, take the node out of the cluster, or reboot the node. Requests to most services are through HAProxy.

For information about how to use Pacemaker for troubleshooting, see Managing high availability services with Pacemaker in the High Availability Deployment and Usage guide.