Chapter 8. Backing Up Satellite Server and Capsule Server

This chapter describes the minimum backup procedure to ensure the continuity of your Red Hat Satellite deployment and associated data in the event of a disaster. If your deployment uses custom configurations, you must consider how to handle these custom configurations when you plan your backup and disaster recovery policy.

8.1. Backing up Satellite Server or Capsule Server

Use this section to create a backup of your Satellite Server or Capsule Server and all associated data using the foreman-maintain backup script. Backing up to a separate storage device on a separate system is highly recommended. Satellite services are unavailable during the backup. The backup can be scheduled for a quiet time using cron, see the A Weekly Full Backup Followed by Daily Incremental Backups.

During offline or snapshot backups, the services are inactive and Satellite is in a maintenance mode. All the traffic from outside on port 443 is rejected by a firewall to ensure there are no modifications triggered.

A backup contains sensitive information from the /root/ssl-build directory. For example, it can contain hostnames, ssh keys, request files and SSL certificates. Either encrypt or move the backup to a secure location to minimize the risk of damage or unauthorized access to the hosts.

Prerequisites

Ensure that no other tasks are scheduled by other administrators for the same time. This is particularly important when administrators are working in different locations and time zones.

Conventional Backup Methods

You can also use conventional backup methods such as that described in the System Backup and Recovery section of the Red Hat Enterprise Linux 7 System Administrator’s Guide. When creating a snapshot or conventional backup, stop all services. Do not do this if using the foreman-maintain backup script:

# foreman-maintain service stop

Start the services after creating a snapshot or conventional backup:

# foreman-maintain service start

8.1.1. Estimating the size of a Backup

The full backup creates uncompressed archives of MongoDB, PostgreSQL, and Pulp database files and Satellite configuration files. Compression occurs after the archives are created to decrease the time when Satellite services are unavailable. Consequently, a full backup requires space to store the following data:

  • Uncompressed Satellite databases and configuration files.
  • Compressed Satellite databases and configuration files.
  • An extra 20% of the total estimated space to ensure a reliable backup.

To Estimate the Size of a Backup

  1. Enter the du command to estimate the size of uncompressed directories containing Satellite databases and configuration files:

    # du -sh /var/lib/mongodb /var/lib/pgsql/data /var/lib/pulp
    480G	/var/lib/mongodb
    100G    /var/lib/pgsql/data
    100G	/var/lib/pulp
    # du -csh /var/lib/qpidd /var/lib/tftpboot /etc /root/ssl-build \
    /var/www/html/pub /opt/puppetlabs
    886M    /var/lib/qpidd
    16M     /var/lib/tftpboot
    37M	/etc
    900K	/root/ssl-build
    100K	/var/www/html/pub
    2M	/opt/puppetlabs
    942M   total
  2. Calculate how much space is required to store the compressed data.

    Table 8.1, “Backup Data Compression Ratio” demonstrates compression ratio of all data items used in the backup.

    Table 8.1. Backup Data Compression Ratio

    Data typeDirectoryRatioExample results

    MongoDB database files

    /var/lib/mongodb

    85 - 90 %

    480 GB → 60 GB

    PostgreSQL database files

    /var/lib/pgsql/data

    80 - 85%

    100 GB → 20 GB

    Pulp RPM files

    /var/lib/pulp

    (not compressed)

    100 GB

    Configuration files

    /var/lib/qpidd
    /var/lib/tftpboot
    /etc
    /root-ssl/build
    /var/www/html/pub
    /opt/puppetlabs

    85%

    942 MB → 141 MB

    In this example, the compressed backup data occupies 180 GB in total.

  3. To calculate how much space you need to store a backup, add the total of the estimated values of compressed and uncompressed backup data and then add an extra 20% to ensure a reliable backup.

    This example requires 681 GB plus 180 GB for the uncompressed and compressed backup data, 861 GB in total. With 172 GB of extra space, 1033 GB must be allocated for the backup location.

8.1.2. Performing a Full Backup of Satellite Server or Capsule Server

Red Hat Satellite 6.4 uses the foreman-maintain backup script to make backups. To see the usage statement, enter the following command:

# foreman-maintain backup --help

There are three main methods of backing up Satellite Server:

  • Offline backup
  • Online backup
  • Snapshot backups

    For more information about each of these methods, you can view the usage statements for each script.

For offline backups:

# foreman-maintain backup offline --help

For online backups:

# foreman-maintain backup online --help

For backups from snapshots:

# foreman-maintain backup snapshot --help

Directory creation

The foreman-maintain backup script creates a time-stamped subdirectory in the backup directory you specify. The foreman-maintain backup script does not overwrite backups. You must select the correct directory or subdirectory when restoring from a backup or an incremental backup. The script stops and restarts services as required.

If you must set the directory name yourself add the --preserve-directory option and add a directory name. The backup is then stored in the directory you provide on the command line.

If you use --preserve-directory, no data is removed if the backup fails.

Note that user postgres needs write access to that directory if you have a local PgSQL database.

Remote databases

You can use the foreman-maintain backup script to back up remote databases.

You can use both online and offline methods to back up remote databases, but if you use offline methods, such as snapshot, the foreman-maintain backup script performs a database dump.

To Perform a Full Offline Backup of Satellite Server or Capsule Server:

Use this procedure to perform a full offline backup. Satellite services are unavailable during the backup process.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure that your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
  2. Run the backup script:

    # foreman-maintain backup offline /var/backup_directory

    This process can take a long time to complete, because of the amount of data to copy.

8.1.3. Performing a Backup without Pulp Content

To Perform a Backup without Pulp Content:

Use this procedure to perform an offline backup but excludes the contents of the Pulp directory. This backup is useful for debugging purposes and is only intended to provide access to configuration files without spending time backing up the Pulp database. You cannot restore from a directory that does not contain Pulp content.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure that your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
  2. Run the backup script:

    # foreman-maintain backup offline --skip-pulp-content /var/backup_directory

8.1.4. Performing an Incremental Backup

To Perform an Incremental Backup:

Use this procedure to perform an offline backup of any changes since a previous backup. Use a full backup as a reference to make the first incremental backup of a sequence. Keep at least the last known good full backup and a complete sequence of incremental backups to restore from.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
  2. Create a full backup:

    # foreman-maintain backup offline  /var/backup_directory
  3. Run the backup script with the --incremental argument. This creates a directory within your backup directory to store the first incremental back up.

    # foreman-maintain backup offline --incremental /var/backup_directory/full_backup  /var/backup_directory
  4. Run the backup script again to create the second incremental backup. Point to your first incremental backup to indicate the starting point for the next increment. This creates a directory for the second incremental backup in your backup directory.

    # foreman-maintain backup offline --incremental /var/backup_directory/first_incremental_backup  /var/backup_directory

    If you want to point to a different version of the backup, and make a series of increments with that version of the backup as the starting point, you can do this at any time. For example, if you want to make a new incremental backup from the full backup rather than the first or second incremental backup, point to the full backup directory:

    # foreman-maintain backup offline --incremental /var/backup_directory/full_backup  /var/backup_directory

8.1.5. Example of a Weekly Full Backup Followed by Daily Incremental Backups

A Weekly Full Backup Followed by Daily Incremental Backups

The script makes a full backup on a Sunday and incremental backups of the following days. Each day that a backup is made, a new subdirectory is created. This script requires a daily cron job.

#!/bin/bash -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/var/backup_directory
if [[ $(date +%w) == 0 ]]; then
  foreman-maintain backup offline --assumeyes $DESTINATION
else
  LAST=$(ls -td -- $DESTINATION/*/ | head -n 1)
  foreman-maintain backup offline --assumeyes --incremental "$LAST" $DESTINATION
fi
exit 0

Note that the foreman-maintain backup script requires /sbin and /usr/sbin directories to be in PATH, and the --assumeyes option, because the command prompts for confirmation for the backup to proceed.

8.1.6. Performing an Online Backup

Perform an online backup only for debugging purposes. If there are procedures affecting the Pulp database, the Pulp part of the backup procedure repeats until it is no longer being altered. Because the backup of the Pulp database is the most time consuming part of backing up Satellite, if you make a change that alters the Pulp database during this time, the backup procedure keeps restarting.

Risks Associated with Online Backups

Data mismatches can occur between Mongo and Postgres databases while the services are online.

When performing an online backup, if there are procedures affecting the Pulp database, the Pulp part of the backup procedure repeats until it is no longer being altered. Because the backup of the Pulp database is the most time consuming part of backing up Satellite, if you make a change that alters the Pulp database during this time, the backup procedure keeps restarting.

For production environments, use the snapshot method. For more information, see Section 8.1.7, “Performing a Snapshot Backup”. If you want to use the online backup method in production, proceed with caution and ensure that no modifications occur during the backup.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

To Perform an Online Backup:

  1. Ensure that your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.
  2. Run the backup script:

    # foreman-maintain backup online /var/backup_directory

8.1.7. Performing a Snapshot Backup

The snapshot backup method uses Logical Volume Manager (LVM) snapshots of the Pulp, MongoDB, and PostgreSQL directories. The backup is then created from the LVM snapshots and not from the running Satellite as with online backup, which mitigates the risk of creating an inconsistent backup. The snapshot backup method is faster than a full offline backup, which reduces Satellite downtime.

To Perform a Snapshot Backup:

To view the usage statement, enter the following command:

foreman-maintain backup snapshot -h

Prerequisites

Before you perform the snapshot backup, ensure that the following conditions exist:

  • The system uses LVM for the directories that you snapshot: /var/lib/pulp/, /var/lib/mongodb/, and /var/lib/pgsql/.
  • The free disk space in the relevant volume group (VG) is three times the size of the snapshot. More precisely, the VG must have enough space unreserved by the member logical volumes (LVs) to accommodate new snapshots. In addition, one of the LVs must have enough free space for the backup directory.
  • The target backup directory is on a different LV than the directories that you snapshot.
Warning

Request other Satellite Server or Capsule Server users to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Run the backup script:

# foreman-maintain backup snapshot /var/backup_directory

The foreman-maintain backup snapshot makes snapshots when the services are active, and stops all services which could impact the backup. This makes the maintenance window shorter. After the successful snapshot, all services are restarted and LVM snapshots are removed.

8.1.8. White-listing and Skipping Steps

A backup using the foreman-maintain backup script proceeds in a sequence of steps. To skip part of the backup add the --whitelist option to the command and add the step label that you want to omit. For example:

# foreman-maintain backup online --whitelist backup-metadata  -y /var/backup_directory

To see a list of available step labels use:

# foreman-maintain advanced procedure run -h