Chapter 8. Backing Up Satellite Server and Capsule Server

You can back up your Satellite deployment to ensure the continuity of your Red Hat Satellite deployment and associated data in the event of a disaster. If your deployment uses custom configurations, you must consider how to handle these custom configurations when you plan your backup and disaster recovery policy.

To create a backup of your Satellite Server or Capsule Server and all associated data, use the satellite-maintain backup command. Backing up to a separate storage device on a separate system is highly recommended.

Satellite services are unavailable during the backup. Therefore, you must ensure that no other tasks are scheduled by other administrators. You can schedule a backup using cron. For more information, see the Section 8.5, “Example of a Weekly Full Backup Followed by Daily Incremental Backups”.

During offline or snapshot backups, the services are inactive and Satellite is in a maintenance mode. All the traffic from outside on port 443 is rejected by a firewall to ensure there are no modifications triggered.

A backup contains sensitive information from the /root/ssl-build directory. For example, it can contain hostnames, ssh keys, request files and SSL certificates. You must encrypt or move the backup to a secure location to minimize the risk of damage or unauthorized access to the hosts.

Conventional Backup Methods

You can also use conventional backup methods. For more information, see System Backup and Recovery in the Red Hat Enterprise Linux 7 System Administrator’s Guide.

Note

If you plan to use the satellite-maintain backup command to create a backup, do not stop the satellite-maintain services.

  • When creating a snapshot or conventional backup, you must stop all services as follows:

    # satellite-maintain service stop
  • Start the services after creating a snapshot or conventional backup:

    # satellite-maintain service start

8.1. Estimating the Size of a Backup

The full backup creates uncompressed archives of MongoDB, PostgreSQL and Pulp database files, and Satellite configuration files. Compression occurs after the archives are created to decrease the time when Satellite services are unavailable.

A full backup requires space to store the following data:

  • Uncompressed Satellite database and configuration files
  • Compressed Satellite database and configuration files
  • An extra 20% of the total estimated space to ensure a reliable backup

Procedure

  1. Enter the du command to estimate the size of uncompressed directories containing Satellite database and configuration files:

    # du -sh /var/lib/mongodb /var/opt/rh/rh-postgresql12/lib/pgsql/data /var/lib/pulp
    480G	/var/lib/mongodb
    100G    /var/opt/rh/rh-postgresql12/lib/pgsql/data
    100G	/var/lib/pulp
    # du -csh /var/lib/qpidd /var/lib/tftpboot /etc /root/ssl-build \
    /var/www/html/pub /opt/puppetlabs
    886M    /var/lib/qpidd
    16M     /var/lib/tftpboot
    37M	/etc
    900K	/root/ssl-build
    100K	/var/www/html/pub
    2M	/opt/puppetlabs
    942M   total
  2. Calculate how much space is required to store the compressed data.

    The following table describes the compression ratio of all data items included in the backup:

    Table 8.1. Backup Data Compression Ratio

    Data typeDirectoryRatioExample results

    MongoDB database files

    /var/lib/mongodb

    85 - 90 %

    480 GB → 60 GB

    PostgreSQL database files

    /var/opt/rh/rh-postgresql12/lib/pgsql/data

    80 - 85%

    100 GB → 20 GB

    Pulp RPM files

    /var/lib/pulp

    (not compressed)

    100 GB

    Configuration files

    /var/lib/qpidd
    /var/lib/tftpboot
    /etc
    /root/ssl-build
    /var/www/html/pub
    /opt/puppetlabs

    85%

    942 MB → 141 MB

    In this example, the compressed backup data occupies 180 GB in total.

  3. To calculate the amount of available space you require to store a backup, calculate the sum of the estimated values of compressed and uncompressed backup data, and add an extra 20% to ensure a reliable backup.

    This example requires 681 GB plus 180 GB for the uncompressed and compressed backup data, 861 GB in total. With 172 GB of extra space, 1033 GB must be allocated for the backup location.

8.2. Performing a Full Backup of Satellite Server or Capsule Server

Red Hat Satellite 6.8 uses the satellite-maintain backup command to make backups.

There are three main methods of backing up Satellite Server:

  • Offline backup
  • Online backup
  • Snapshot backups

    For more information about each of these methods, you can view the usage statements for each backup method.

For offline backups:

# satellite-maintain backup offline --help

For online backups:

# satellite-maintain backup online --help

For snapshots backups:

# satellite-maintain backup snapshot --help

Directory creation

The satellite-maintain backup command creates a time-stamped subdirectory in the backup directory that you specify. The satellite-maintain backup command does not overwrite backups, therefore you must select the correct directory or subdirectory when restoring from a backup or an incremental backup. The satellite-maintain backup command stops and restarts services as required.

When you run the satellite-maintain backup offline command, the following default backup directories are created:

  • satellite-backup on Satellite
  • foreman-proxy-backup on Capsule

If you want to set a custom directory name, add the --preserve-directory option and add a directory name. The backup is then stored in the directory you provide in the command line. If you use the --preserve-directory option, no data is removed if the backup fails.

Note that if you use a local PgSQL database, the postgres user requires write access to the backup directory.

Remote databases

You can use the satellite-maintain backup command to back up remote databases.

You can use both online and offline methods to back up remote databases, but if you use offline methods, such as snapshot, the satellite-maintain backup command performs a database dump.

Prerequisites

Procedure

To perform a full offline backup of Satellite Server or Capsule Server, complete one of the following steps:

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  • On Satellite Server, enter the following command:

    # satellite-maintain backup offline /var/satellite-backup
  • On Capsule Server, enter the following command:

    # satellite-maintain backup offline /var/foreman-proxy-backup

8.3. Performing a Backup without Pulp Content

You can perform an offline backup that excludes the contents of the Pulp directory. The backup without Pulp content is useful for debugging purposes and is only intended to provide access to configuration files without backing up the Pulp database. You cannot restore from a directory that does not contain Pulp content.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Prerequisites

Procedure

  • To perform an offline backup without Pulp content, enter the following command:

    # satellite-maintain backup offline --skip-pulp-content /var/backup_directory

8.4. Performing an Incremental Backup

Use this procedure to perform an offline backup of any changes since a previous backup.

To perform incremental backups, you must perform a full backup as a reference to create the first incremental backup of a sequence. Keep the most recent full backup and a complete sequence of incremental backups to restore from.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Prerequisites

Procedure

  1. To perform a full offline backup, enter the following command:

    # satellite-maintain backup offline  /var/backup_directory
  2. To create a directory within your backup directory to store the first incremental back up, enter the satellite-maintain backup command with the --incremental option:

    # satellite-maintain backup offline --incremental /var/backup_directory/full_backup  /var/backup_directory
  3. To create the second incremental backup, enter the satellite-maintain backup command with the --incremental option and include the path to the first incremental backup to indicate the starting point for the next increment. This creates a directory for the second incremental backup in your backup directory:

    # satellite-maintain backup offline --incremental /var/backup_directory/first_incremental_backup  /var/backup_directory
  4. Optional: If you want to point to a different version of the backup, and make a series of increments with that version of the backup as the starting point, you can do this at any time. For example, if you want to make a new incremental backup from the full backup rather than the first or second incremental backup, point to the full backup directory:

    # satellite-maintain backup offline --incremental /var/backup_directory/full_backup  /var/backup_directory

8.5. Example of a Weekly Full Backup Followed by Daily Incremental Backups

The following script performs a full backup on a Sunday followed by incremental backups for each of the following days. A new subdirectory is created for each day that an incremental backup is performed. The script requires a daily cron job.

#!/bin/bash -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/var/backup_directory
if [[ $(date +%w) == 0 ]]; then
  satellite-maintain backup offline --assumeyes $DESTINATION
else
  LAST=$(ls -td -- $DESTINATION/*/ | head -n 1)
  satellite-maintain backup offline --assumeyes --incremental "$LAST" $DESTINATION
fi
exit 0

Note that the satellite-maintain backup command requires /sbin and /usr/sbin directories to be in PATH and the --assumeyes option is used to skip the confirmation prompt.

8.6. Performing an Online Backup

Perform an online backup only for debugging purposes.

Risks Associated with Online Backups

Data mismatches can occur between Mongo and Postgres databases while the services are online.

When performing an online backup, if there are procedures affecting the Pulp database, the Pulp part of the backup procedure repeats until it is no longer being altered. Because the backup of the Pulp database is the most time consuming part of backing up Satellite, if you make a change that alters the Pulp database during this time, the backup procedure keeps restarting.

For production environments, use the snapshot method. For more information, see Section 8.7, “Performing a Snapshot Backup”. If you want to use the online backup method in production, proceed with caution and ensure that no modifications occur during the backup.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Prerequisites

Procedure

  • To perform an online backup, enter the following command:

    # satellite-maintain backup online /var/backup_directory

8.7. Performing a Snapshot Backup

You can perform a snapshot backup that uses Logical Volume Manager (LVM) snapshots of the Pulp, MongoDB, and PostgreSQL directories. Creating a backup from LVM snapshots mitigates the risk of an inconsistent backup.

The snapshot backup method is faster than a full offline backup and therefore reduces Satellite downtime.

To view the usage statement, enter the following command:

satellite-maintain backup snapshot -h
Warning

Request other Satellite Server or Capsule Server users to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Prerequisites

Before you perform the snapshot backup, ensure that the following conditions are met:

  • The system uses LVM for the directories that you snapshot: /var/lib/pulp/, /var/lib/mongodb/, and /var/lib/pgsql/.
  • The free disk space in the relevant volume group (VG) is three times the size of the snapshot. More precisely, the VG must have enough space unreserved by the member logical volumes (LVs) to accommodate new snapshots. In addition, one of the LVs must have enough free space for the backup directory.
  • The target backup directory is on a different LV than the directories that you snapshot.

Procedure

  • To perform a snapshot backup, enter the satellite-maintain backup snapshot command:

    # satellite-maintain backup snapshot /var/backup_directory

The satellite-maintain backup snapshot command creates snapshots when the services are active, and stops all services which can impact the backup. This makes the maintenance window shorter. After the successful snapshot, all services are restarted and LVM snapshots are removed.

8.8. White-listing and Skipping Steps When Performing Backups

A backup using the satellite-maintain backup command proceeds in a sequence of steps. To skip part of the backup add the --whitelist option to the command and add the step label that you want to omit.

  • To display a list of available step labels, enter the following command:

    # satellite-maintain advanced procedure run -h
  • To skip a step of the backup, enter the satellite-maintain backup command with the --whitelist option. For example:

    # satellite-maintain backup online --whitelist backup-metadata  -y /var/backup_directory