Red Hat Training

A Red Hat training course is available for Red Hat Satellite

Chapter 8. Backing Up and Restoring Satellite Server and Capsule Server

This chapter describes the minimum backup and restore procedures required to ensure continuity of your Red Hat Satellite deployment and associated data in the event of a disaster. If your deployment uses custom configurations you should take these into account when planning your backup and disaster recovery policy.

8.1. Backing up Satellite Server or Capsule Server

This section describes creating a backup of your Satellite Server or Capsule Server and all associated data using the satellite-backup script. Backing up to a separate location is recommended. Backing up to a separate storage device on a separate system is highly recommended. Satellite services are unavailable during the backup. The backup can be scheduled for a quiet time using cron, see the Example 8.1, “A Weekly Full Backup Followed by Daily Incremental Backups”.

Prerequisites

  • When planning a scheduled backup, ensure that no other tasks are scheduled by other administrators for the same time. This is particularly important when administrators are working in different locations and time zones.
  • Ensure that you either encrypt the backups or move them to a secure location to minimize the risk of damage or unauthorized access to hosts.
Note

You can also use conventional backup methods such as that described in the System Backup and Recovery section of the Red Hat Enterprise Linux 7 System Administrator’s Guide. When creating a snapshot or conventional backup, stop all services (Do not do this if using the satellite-backup script): 

# katello-service stop

Start the services after creating a snapshot or conventional backup: 

# katello-service start

8.1.1. Estimating the size of a Backup

The full backup creates uncompressed archives of MongoDB, PostgreSQL, and Pulp database files and Satellite configuration files. Compression occures after the archives are created to decrease the time when Satellite services are unavailable. Consequently, a full backup requires space to store the following data:

  • Uncompressed Satellite databases and configuration files.
  • Compressed Satellite databases and configuration files.
  • An extra 20% of the total estimated space to ensure a reliable backup.

To Estimate the Size of a Backup

  1. Enter the du command to estimate the space required to store uncompressed directories containing Satellite databases and configuration files, for example: 

    # du -csh /var/lib/mongodb /var/lib/pgsql/data /var/lib/pulp \
/etc /root/ssl-build /var/www/html/pub /opt/puppetlabs
480G	/var/lib/mongodb
100G    /var/lib/pgsql/data
100G	/var/lib/pulp
680G    total
37M	/etc
900K	/root/ssl-build
100K	/var/www/html/pub
2M	/opt/puppetlabs
680GB   total

    In this example, the uncompressed backup data occupies 680 GB in total.

    Note

    The /opt/puppetlabs directory is used for Puppet 4. Use /var/lib/puppet for Puppet 3.

  2. Calculate how much space is required to store the compressed data.

    Table 8.1, “Backup Data Compression Ratio” demonstrates compression ratio of all data items used in the backup.

    Table 8.1. Backup Data Compression Ratio

    Data typeDirectoryRatioExample results

    MongoDB database files

    /var/lib/mongodb

    10 - 15%

    480 GB → 420 GB

    PostgreSQL database files

    /var/lib/pgsql/data

    15 - 20%

    100 GB → 80 GB

    Pulp RPM files

    /var/lib/pulp

    -

    (not compressed)

    Configuration files

    /etc
    /root-ssl/build
    /var/www/html/pub
    /opt/puppetlabs

    5 - 10%

    50 MB → 45 MB

    In this example, the compressed backup data occupies 500 GB in total.

  3. Calculate how much space is required to store a backup. Sum up the estimated values of compressed and uncompressed backup data and add extra 20% to ensure a reliable backup.

    This example requires 680 GB plus 500 GB for the uncompressed and compressed backup data, 1180 GB in total. With 240 GB of extra space, 1420 GB must be allocated for the backup location.

8.1.2. Performing a Full Backup of Satellite Server or Capsule Server

Red Hat Satellite 6.3 uses the satellite-backup script to make and restore backups. To see the usage statement, enter a command as follows: 

# satellite-backup --help

From Satellite 6.2.8, the satellite-backup creates a time-stamped subdirectory in the backup directory you specify. The satellite-backup script does not overwrite backups and the correct directory or subdirectory has to be selected when restoring from a backup or an incremental backup. The script stops and restarts services as required.

To Perform a Full Offline Backup of Satellite Server or Capsule Server:

This procedure performs a full offline backup. Satellite services are unavailable during the backup process.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.

  2. Run the backup script: 

    # satellite-backup backup_directory

    The satellite-backup script stops all services which could impact the backup, performs the backup, then restarts the required services. The script creates the target directory when trying to create a backup file if the target directory does not exist.

    This process can take a long time to complete, due to the amount of data to copy.

8.1.3. Performing a Backup without Pulp Content

To Perform a Backup without Pulp Content:

This procedure performs an off-line backup but excludes the contents of the Pulp directory. This backup is useful for debugging purposes and is only intended to provide access to configuration files without spending time backing up the Pulp database. You cannot restore from a directory that does not contain Pulp content.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.

  2. Run the backup script: 

    # satellite-backup --skip-pulp-content backup_directory

    The satellite-backup script stops all services which could impact the backup, performs the backup, then restarts the required services. The script creates the target directory when trying to create a backup file if the target directory does not exist.

8.1.4. Performing an Incremental Backup

To Perform an Incremental Backup:

This procedure performs an off-line backup of any changes since a previous backup. Use a full backup as a reference to make the first incremental backup of a sequence. Keep at least the last known good full backup and a complete sequence of incremental backups to restore from.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.

  2. Run the backup script:

    With Pulp content: 

    # satellite-backup backup_directory --incremental backup_directory/previous_time-stamped_subdirectory

    Without Pulp content: 

    # satellite-backup backup_directory --skip-pulp-content --incremental backup_directory/previous_time-stamped_subdirectory

    The satellite-backup script stops all services which could impact the backup, performs the backup, then restarts the required services. If the target directory does not exist when trying to create a backup file the script creates it. It is possible to make incremental backups using a backup older than the previous backup as a starting point, but with a corresponding increase in time to make the backup.

8.1.5. Example - A Weekly Full Backup Followed by Daily Incremental Backups

Example 8.1. A Weekly Full Backup Followed by Daily Incremental Backups

An example script that makes a full backup on a Sunday and incremental backups on all other days of the week. Backups are stored in subdirectories named $YEAR-$WEEK. Each subdirectory contains the full backup from the previous week’s Sunday and incremental backups from the subsequent days. This script requires a daily cron job. 

#!/bin/bash -e
export PATH=/sbin:/bin:/usr/sbin:/usr/bin
DESTINATION=/var/backup
YEAR=$(date +%Y)
WEEK=$(date +%-V)
if [[ $(date +%w) == 0 ]]; then
  satellite-backup $DESTINATION/$YEAR-$((WEEK + 1)) --assumeyes
else
  LAST=$(ls -td -- $DESTINATION/$YEAR-$WEEK/*/ | head -n 1)
  satellite-backup $DESTINATION/$YEAR-$WEEK --incremental "$LAST" --assumeyes
fi
exit 0

Note that the satellite-backup script requires /sbin and /usr/sbin directories to be in PATH.

8.1.6. Performing an Online Backup

To Perform an Online Backup:

This procedure performs a full backup while Satellite Server or Capsule Server is running. If there are procedures affecting the Pulp database, the Pulp part of the backup procedure repeats until it is no longer being altered. Since the backup of the Pulp database is the most time consuming part of backing up a Satellite, it is highly recommended to not alter the Pulp database during this time. This prolongs the backup procedure as the Pulp part of the backup is restarted.

Important

Satellite 6 uses two database systems, PostgreSQL and MongoDB. There are records that exist in both PostgreSQL and MongoDB that need to remain synchronized.

The --online-backup option keeps all services running which means there is a possibility that data can be modified while the backup is being made. There is a basic check to see if the databases were modified during the backup. If this occurs, the script starts the database portion of the backup again. This check is rudimentary and cannot ensure with 100% certainty that there were no modifications to the databases while the backup script was running. This check can also result in repeated loops if there is continuous modification occurring to the databases.

It is recommended to use a snapshot backup method in production, as described in Section 8.1.7, “Performing a Snapshot Backup”. If you still want to use the online backup method in production, ensure that no modifications occur during the backup.

Warning

Request other users of Satellite Server or Capsule Server to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

  1. Ensure your backup location has enough disk space to store the backup. For more information, see Section 8.1.1, “Estimating the size of a Backup”.

  2. Run the backup script: 

    # satellite-backup --online-backup /tmp/backup_directory

8.1.7. Performing a Snapshot Backup

The snapshot backup method uses Logical Volume Manager (LVM) snapshots of the Pulp, MongoDB, and PostgreSQL directories. The actual backup is then created from the LVM snapshots and not from the running Satellite as with online backup, which mitigates the risk of creating an inconsistent backup. The snapshot backup method is faster than a full off-line backup, which reduces Satellite downtime. As such, it suits well for backing up highly populated Satellite servers with long backup times.

To Perform a Snapshot Backup:

This procedure performs a snapshot backup. It can be combined with other satellite-backup sub-commands, except an online-backup.

Prerequisites

  • The system uses LVM for the snapshotted directories (/var/lib/pulp/, /var/lib/mongodb/, and /var/lib/pgsql/).
  • The free disk space in the relevant volume group (VG) is three times the size of the snapshot. More precisely, the VG has to have enough space unreserved by the member logical volumes (LVs) to accommodate new snapshots. In addition, one of the LVs has to have enough free space for the backup directory.
  • The target backup directory is on a different LV than the snapshotted directories.
Warning

Request other Satellite Server or Capsule Server users to save any changes and warn them that Satellite services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

Run the backup script: 

# satellite-backup --snapshot backup_directory

The satellite-backup script stops all services which could impact the backup. After the successful backup, all services are restarted and LVM snapshots are removed.

8.2. Restoring Satellite Server or Capsule Server from a Backup

This section describes how to restore a Red Hat Satellite Server or Red Hat Capsule Server from the backup data created as a result of following the steps in Section 8.1, “Backing up Satellite Server or Capsule Server”. This process is intended for restoring the backup on the same server that generated the backup, and all data covered by the backup is deleted on the target system. If the original system is unavailable, provision a system with the same configuration settings (in particular, the host name must be the same).

Prerequisites

  • Ensure that you are restoring to the correct instance. The Red Hat Satellite instance must have the same host name, configuration, and be the same major version as the original system.
  • Ensure that you run the satellite-restore script as root.

  • Ensure that all SELinux contexts are correct. Enter the following command to restore the correct SELinux contexts: 

    # restorecon -Rnv /

To Restore Satellite Server or Capsule Server from a Full Backup:

  1. Install Satellite 6 using the procedures in Installing Satellite Server in the Installation Guide.

  2. Copy the backup data to the Satellite’s local file system. Use /tmp/ or /var/tmp/. Ensure you have enough space to store this data on the base system of Satellite Server or Capsule Server as well as enough space after the restoration to contain all the data in the /etc/ and /var/ directories contained within the backup.

    You can use the du -sh directory_name command to check the space used by a directory and the df -h directory_name command to check for free space. Add the --total option to sum the results from more than one directory.

  3. Run the restoration script: 

    # satellite-restore backup_directory

    Where backup_directory is the time-stamped directory or subdirectory containing the backed-up data. The target directory is read from the configuration files contained within the archive. If the target directory does not exist when trying to recover, it gives an error and asks for the correct directory. The restore process can take a long time to complete, due to the amount of data to copy. Where incremental backups exist, see To Restore Satellite Server or Capsule Server from an Incremental Backup:.

When this process completes, all services should be running and Satellite Server or Capsule Server should be available for use.

To Restore Satellite Server or Capsule Server from an Incremental Backup:

  1. Install Satellite 6 using the procedures in Installing Satellite Server in the Installation Guide.
  2. Restore the last full backup as described in To Restore Satellite Server or Capsule Server from a Full Backup:.
  3. Copy the backup data to the Satellite’s local file system, for example, /var/tmp/satellite-backup/. Ensure you have enough space to store this data on the base system of Satellite Server or Capsule Server as well as enough space after the restoration to contain all the data in the /etc/ and /var/ directories contained within the backup.

  4. Run the restoration script: 

    # satellite-restore backup_directory_X

    Where backup_directory_X is a time-stamped directory or subdirectory containing an incremental backup. Restore the incremental backups in the same sequence that they were made. For example: backup_directory_1, backup_directory_2. The target directory is read from the configuration files contained within the archive. If the target directory does not exist when trying to recover, it gives an error and asks for the correct directory.

When this process completes, all services should be running and Satellite Server or Capsule Server should be available for use.

8.3. Backing up and Restoring Capsule Server Using a Virtual Machine Snapshot

There are three methods of backing up Capsule Server:

  • Using the satellite-backup script as described in Section 8.1, “Backing up Satellite Server or Capsule Server”. The satellite-backup script is convenient if your Capsule Server is a physical machine. You can also use the script if your Capsule Server is a virtual machine but it creates only a backup of the data and not the machine itself.
  • Using the conventional backup methods as described in System Backup and Recovery in the Red Hat Enterprise Linux 7 System Administrator’s Guide.
  • Using the snapshot of a virtual machine with your Capsule Server on it as described below. Note that this method is different from the snapshot backup method described in Section 8.1.7, “Performing a Snapshot Backup”.

If your Capsule Server is a virtual machine, you can restore it from a snapshot. Creating weekly snapshots to restore from is recommended. In the event of failure, you can reinstall, or configure a new Capsule Server, and then synchronize the database content from the Satellite Server.

Note

When creating a snapshot or conventional backup, stop all services (Do not do this if using the satellite-backup script): 

# katello-service stop

Start the services after creating a snapshot or conventional backup: 

# katello-service start

If you have a snapshot or conventional backup, restore from that and then synchronize from the Satellite Server as described below.

If required, deploy a new Capsule Server, ensuring the host name is the same as before, and then install the Capsule certificates. You might still have them on the Satellite Server, the package name ends in -certs.tar, alternately create new ones. Follow the procedures in Installing Capsule Server in the Installation Guide until you see in the web UI that the Capsule Server is connected to the Satellite Server. Then synchronize from the Satellite Server as described below.

Synchronizing an External Capsule

  1. To synchronize an external Capsule, select the relevant organization and location in the web UI, or choose Any Organization and Any Location.
  2. Navigate to Infrastructure > Capsules and click the name of the Capsule to synchronize.
  3. On the Overview tab, select Synchronize.

8.4. Renaming a Satellite Server or Capsule Server

Renaming a Satellite Server or Capsule Server requires use of the satellite-change-hostname script. Red Hat Satellite contains references to the host’s name and these changes are made using the script. Renaming a Satellite Server affects itself, all Capsule Servers and all hosts registered to it. Renaming a Capsule Server affects itself and all hosts registered to it.

Warning

The renaming process shuts down all Satellite Server services on the host being renamed. When the renaming is complete, all services are restarted.

8.4.1. Renaming a Satellite Server

The host name of a Satellite Server is used by Satellite Server components, all Capsule Servers, and hosts registered to it for communication. Renaming a Satellite Server requires that these references be updated.

If you use external authentication, you must reconfigure Satellite Server for external authentication after you run the satellite-change-hostname script. The satellite-change-hostname script breaks external authentication for Satellite Server. For more information about configuring external authentication, see Chapter 11, Configuring External Authentication

Prerequisites

  • (Optional) If the Satellite Server has a custom X.509 certificate installed, a new certificate must be obtained in the host’s new name. When all hosts are re-registered to the Satellite Server, the new certificate is installed. For more information on obtaining a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.
  • Backup the Satellite Server. The satellite-change-hostname script makes irreversible changes to the Satellite Server. If the renaming process is not successful, you must restore it from backup. For more information, see Section 11.2, “Using Identity Management”.

Rename a Satellite Server

  1. On the Satellite Server, run the satellite-change-hostname script, providing the host’s new name, and Satellite credentials. 

    # satellite-change-hostname new_satellite --username admin \
--password password

    The message ***** Hostname change complete! ***** confirms that the rename completed successfully.

  2. (Optional) If you have obtained a new X.509 certificate for the Satellite Server’s new host name, run the Satellite installation script to install the certificate. For more information on installing a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.

  3. On all Capsule Servers and hosts registered to the Satellite Server, reinstall the bootstrap RPM and re-register them to the Satellite Server. Substitute the example organization and environment values with those matching your environment.

    1.   

      # yum remove -y katello-ca-consumer*

    2.   

      # rpm -Uvh http://new-satellite.example.com/pub/katello-ca-consumer-latest.noarch.rpm

    3.   

      # subscription-manager register --org="Default_Organization" \
--environment="Library" \
--force

    Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Configuring and Running Remote Commands in Managing Hosts.

  4. Reattach subscriptions to all Capsule Servers and hosts registered to the Satellite Server, then refresh the subscription.

    1.   

      # subscription-manager refresh

    2.   

      # yum repolist

    Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Configuring and Running Remote Commands in Managing Hosts.

  5. On all Capsule Servers, re-run the Satellite installation script to update references to the new host name. 

    # satellite-installer --foreman-proxy-content-parent-fqdn new-satellite.example.com \
--foreman-proxy-foreman-base-url  https://new-satellite.example.com \
--foreman-proxy-trusted-hosts new-satellite.example.com

  6. On the Satellite Server, synchronize content for each Capsule Server.

    1. List all Capsule Servers with their ID numbers: 

      # hammer capsule list

    2. Enter the following command for each Capsule Server: 

      # hammer capsule content synchronize --id capsule_id_number

8.4.2. Renaming a Capsule Server

The host name of a Capsule Server is referenced by Satellite Server components, and all hosts registered to it. Renaming a Capsule Server requires that these references be updated.

Prerequisites

Renaming a Capsule Server:

  1. On Satellite Server, create a new certificates archive file.

    1. If you are using the default Satellite Server certificate: 

      # capsule-certs-generate --capsule-fqdn new-capsule.example.com \
--certs-tar /root/new-capsule.example.com-certs.tar

      Ensure that you enter the full path to the .tar file.

    2. If you are using a custom X.509 certificate on the Capsule Server, see Create the Capsule Server’s Certificate Archive File in the Red Hat Satellite Installation Guide.

  2. On Satellite Server, copy the certificates archive file to the Capsule Server, providing the root user’s password when prompted. In this example the archive file is copied to the root user’s home directory, but you may prefer to copy it elsewhere. 

    # scp /root/new-capsule.example.com-certs.tar root@capsule.example.com:

  3. On the Capsule Server, run the satellite-change-hostname script, providing the host’s new name, Satellite credentials, and certificates archive filename. 

    # satellite-change-hostname new_capsule --username admin \
--password password \
--certs-tar /root/new-capsule.example.com-certs.tar

    Ensure that you enter the full path to the .tar file.

    The message ***** Hostname change complete! ***** confirms that the rename completed successfully.

  4. Optional: If you have obtained a new X.509 certificate in the Capsule Server’s new host name, run the Satellite installation script to install the certificate. For more information on installing a custom X.509 certificate, see Configuring Satellite Server with a Custom Server Certificate in the Red Hat Satellite Installation Guide.

  5. On all hosts registered to the Capsule Server, reinstall the bootstrap RPM and re-register them to the Capsule Server. Substitute the example organization and environment values with those matching your environment. 

    # yum remove -y katello-ca-consumer*
    # rpm -Uvh http://new-capsule.example.com/pub/katello-ca-consumer-latest.noarch.rpm
    # subscription-manager register --org="Default_Organization" \
--environment="Library" \
--force

    Use of the Red Hat Satellite remote execution feature is recommended for this step. For more information, see Running Jobs on Hosts in Managing Hosts.

  6. Reattach subscriptions to all hosts registered to the Capsule Server, then refresh the subscription. 

    # subscription-manager refresh
    # yum repolist

  7. Edit the Capsule Server’s name.

    1. In the Satellite web UI, navigate to Infrastructure > Capsules.
    2. Find the Capsule Server in the list, and click Edit in that row.
    3. Edit the Name and URL fields to match the Capsule Server’s new host name, then click Submit.
  8. On your DNS server, add a record for the Capsule Server’s new host name, and delete the record for the previous host name.