⁠Appendix A. Appendix

The following sections provide supplementary information for this guide.

⁠A.1. Backup and Restore

Both a backup and a recovery policy should be formulated to minimize data loss and system downtime.
When determining your backup strategy, you will need to answer the following questions:
  • How quickly will you need to recover from data loss? If you cannot have data loss at all, you should focus on High Availability as a deployment strategy, in addition to using backups.
  • How many backups should you keep?
  • Should your backups be kept off-site?
  • How often should backups be tested?
  • What will be backed up? Critical data in OpenStack is backed up per component (or service).
The following sections describe database and file-system backups for components, as well as information on recovering backups.

⁠A.1.1. Database Backup

The installation process in this guide uses the MariaDB server, which hosts the databases for the separate OpenStack components. With all these databases in one place, it is easy to create a database backup.

Note

If you are using a different database server, refer to your server's notes for more information. For example, if you are using MariaDB and MylSAM tables, consider using mysqlhotcopy to improve backup and recovery times.
⁠

Procedure A.1. Backing up all databases

  1. Log in as the database root user.
  2. Export database information into a backup file: 
    # mysqldump --opt --all-databases > fileName
    For example: 
    # mysqldump --opt --all-databases > openstack.sql
⁠

Procedure A.2. Backing up a single database

  1. Log in as the database root user.
  2. Export a single database into a backup file: 
    # mysqldump --opt databaseName > fileName
    For example: 
    # mysqldump --opt nova > nova.sql

Note

You can automate the backup process by creating a cron job that runs the following script once per day: 
#!/bin/bash
backup_dir="/var/lib/backups/mysql"
filename="${backup_dir}/mysql-`hostname`-`eval date +%Y%m%d`.sql.gz"

# Dump the entire MariaDB database
/usr/bin/mysqldump --opt --all-databases | gzip > $filename 

# Delete backups older than 7 days
find $backup_dir -ctime +7 -type f -name mysql-\*sql.gz -delete

⁠A.1.2. File System Backup

Each OpenStack component has its own directory and files under /etc, /var/lib, and /var/log. Files in these directories should be regularly backed up:
/etc/component
Files in this directory (for example, /etc/glance) should be regularly backed up. This will be the case for all nodes on which the component is housed. In particular:
  • /etc/nova will need to be backed up on both the cloud controller and all Compute nodes.
  • /etc/swift is very important, because it contains Object Storage configuration files, ring files, and ring builder files. Losing these files will render the data on your cluster inaccessible. A best practice is to copy the builder files to all storage nodes along with the ring files. Try to spread multiple backup copies throughout your storage cluster.
/var/lib/component
It is good practice to back up this directory (for example, /var/lib/keystone) for all components. In particular:
  • /var/lib/glance/images - If you are using a file-based backend for the Image service, this directory stores its images and should be backed up regularly. Using both of the following should ensure image availability:
    • Run the directory on a RAID array; if a disk fails, the directory is available.
    • Use a tool such as rsync to replicate the images to another server: 
      # rsync -az --progress /var/lib/glance/images backup-server: \
/var/lib/glance/images/
  • /var/lib/nova - All files in this directory should be backed up. The only exception is the /var/lib/nova/instances subdirectory on Compute nodes. This subdirectory contains the KVM images of running instances. You would only want to back up this directory if you need to maintain backup copies of all instances. Under most circumstances, you do not need to do this, but this can vary from cloud to cloud and your service levels. Also, be aware that making a backup of a live KVM instance can cause that instance to not boot properly if it is ever restored from a backup.
/var/log/component
Log files should be regularly backed up, unless you have all logs going to a central server. It is highly recommended to use a central logging server.

Note

Various tools can be used to backup files. Red Hat Enterprise Linux supports the following:
  • tar
    For example, to create an archive file called var-lib-cinder-backup.tar in /mnt/backup/, run: 
    # tar cf /mnt/backup/var-lib-cinder-backup.tar /var/lib/cinder
    The resulting archive file contains the contents of the /var/lib/cinder directory, and will be nearly as large as the data being backed up. Depending on the type of data being backed up, compressing the archive file can result in significant size reductions. The archive file can be compressed by adding a single option to the previous command: 
    # tar czf /mnt/backup/var-lib-cinder-backup.tar.gz /var/lib/cinder
    The resulting var-lib-cinder-backup.tar.gz archive file is now gzip compressed.
  • cpio
    Unlike tar, cpio reads the names of the files it processes through standard input. For example: 
    # find /var/lib/cinder/ | cpio -o > /mnt/backup/var-lib-cinder-backup.cpio
    This command creates a cpio archive file (containing everything in /var/lib/cinder/) called var-lib-cinder-backup.cpio and places it in the /mnt/backup/ directory.

⁠A.1.3. Recovery

Recovery involves the simple procedure of stopping services, restoring files and databases for the component, then restarting the services. All procedures must be carried out as the root user.
⁠

Procedure A.3. Recover database backups and files

  1. Ensure all services are stopped for the component. For example, for the Image service: 
    # service openstack-glance-registry stop
# service openstack-glance-api stop
  2. Stop MariaDB: 
    # service mariadb stop
  3. Import the previously backed-up database. For example, to import an Image service back up: 
    # mysql glance < glance.sql
  4. Copy backup files over to the necessary directories. For example, for Image files: 
    # mv /etc/glance{,.orig}
# cp -a /path/to/backup/glance /etc/
  5. Restart the component's services. For example, to restart the Image service you will have to restart the SQL database, Image registry, and Image API. 
    # service mariadb start
# service openstack-glance-registry start
# service openstack-glance-api start