Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 8. Backing Up and Restoring IdM

Red Hat Enterprise Linux Identity Management provides a solution to manually back up and restore the IdM system. This may be necessary after a data loss event.

During backup, the system creates a directory containing information about your IdM setup and stores it. During restore, you can use this backup directory to bring your original IdM setup back.

Note

The IdM backup and restore features are designed to help prevent data loss. To mitigate the impact of losing a server, and ensure continued operation by providing alternative servers to clients, ensure you have a replica topology according to Mitigating server loss with replication.

8.1. IdM backup types

With the ipa-backup utility, you can create two types of backups:

Full-server backup
  • Contains all server configuration files related to IdM, and LDAP data in LDAP Data Interchange Format (LDIF) files
  • IdM services must be offline.
  • Suitable for rebuilding an IdM deployment from scratch.
Data-only backup
  • Contains LDAP data in LDIF files and the replication changelog
  • IdM services can be online or offline.
  • Suitable for restoring IdM data to a state in the past

8.2. Naming conventions for IdM backup files

By default, IdM stores backups as .tar archives in subdirectories of the /var/lib/ipa/backup/ directory.

The archives and subdirectories follow these naming conventions:

Full-server backup

An archive named ipa-full.tar in a directory named ipa-full-<YEAR-MM-DD-HH-MM-SS>, with the time specified in GMT time.

[root@server ~]# ll /var/lib/ipa/backup/ipa-full-2021-01-29-12-11-46
total 3056
-rw-r--r--. 1 root root     158 Jan 29 12:11 header
-rw-r--r--. 1 root root 3121511 Jan 29 12:11 ipa-full.tar
Data-only backup

An archive named ipa-data.tar in a directory named ipa-data-<YEAR-MM-DD-HH-MM-SS>, with the time specified in GMT time.

[root@server ~]# ll /var/lib/ipa/backup/ipa-data-2021-01-29-12-14-23
total 1072
-rw-r--r--. 1 root root     158 Jan 29 12:14 header
-rw-r--r--. 1 root root 1090388 Jan 29 12:14 ipa-data.tar
Note

Uninstalling an IdM server does not automatically remove any backup files.

8.3. Considerations when creating a backup

The important behaviors and limitations of the ipa-backup command include the following:

  • By default, the ipa-backup utility runs in offline mode, which stops all IdM services. The utility automatically restarts IdM services after the backup is finished.
  • A full-server backup must always run with IdM services offline, but a data-only backup may be performed with services online.
  • By default, the ipa-backup utility creates backups on the file system containing the /var/lib/ipa/backup/ directory. Red Hat recommends creating backups regularly on a file system separate from the production filesystem used by IdM, and archiving the backups to a fixed medium, such as tape or optical storage.
  • Consider performing backups on hidden replicas. IdM services can be shut down on hidden replicas without affecting IdM clients.
  • Starting with RHEL 8.3.0, the ipa-backup utility checks if all of the services used in your IdM cluster, such as a Certificate Authority (CA), Domain Name System (DNS), and Key Recovery Agent (KRA), are installed on the server where you are running the backup. If the server does not have all these services installed, the ipa-backup utility exits with a warning, because backups taken on that host would not be sufficient for a full cluster restoration.

    For example, if your IdM deployment uses an integrated Certificate Authority (CA), a backup run on a non-CA replica will not capture CA data. Red Hat recommends verifying that the replica where you perform an ipa-backup has all of the IdM services used in the cluster installed.

    You can bypass the IdM server role check with the ipa-backup --disable-role-check command, but the resulting backup will not contain all the data necessary to restore IdM fully.

8.4. Creating an IdM backup

Follow this procedure to create a full-server and data-only backup in offline and online modes using the ipa-backup command.

Prerequisites

  • You must have root privileges to run the ipa-backup utility.

Procedure

  • To create a full-server backup in offline mode, use the ipa-backup utility without additional options.

    [root@server ~]# ipa-backup
    Preparing backup on server.example.com
    Stopping IPA services
    Backing up ipaca in EXAMPLE-COM to LDIF
    Backing up userRoot in EXAMPLE-COM to LDIF
    Backing up EXAMPLE-COM
    Backing up files
    Starting IPA service
    Backed up to /var/lib/ipa/backup/ipa-full-2020-01-14-11-26-06
    The ipa-backup command was successful
  • To create an offline data-only backup, specify the --data option.

    [root@server ~]# ipa-backup --data
  • To create a full-server backup that includes IdM log files, use the --logs option.

    [root@server ~]# ipa-backup --logs
  • To create a data-only backup while IdM services are running, specify both --data and --online options.

    [root@server ~]# ipa-backup --data --online
Note

If the backup fails due to insufficient space in the /tmp directory, use the TMPDIR environment variable to change the destination for temporary files created by the backup process:

[root@server ~]# TMPDIR=/new/location ipa-backup

For more details, see ipa-backup Command Fails to Finish.

Verification Steps

  • The backup directory contains an archive with the backup.

    [root@server ~]# ls /var/lib/ipa/backup/ipa-full-2020-01-14-11-26-06
    header  ipa-full.tar

8.5. Creating a GPG2-encrypted IdM backup

You can create encrypted backups using GNU Privacy Guard (GPG) encryption. The following procedure creates an IdM backup and encrypts it using a GPG2 key.

Prerequisites

Procedure

  • Create a GPG-encrypted backup by specifying the --gpg option.

    [root@server ~]# ipa-backup --gpg
    Preparing backup on server.example.com
    Stopping IPA services
    Backing up ipaca in EXAMPLE-COM to LDIF
    Backing up userRoot in EXAMPLE-COM to LDIF
    Backing up EXAMPLE-COM
    Backing up files
    Starting IPA service
    Encrypting /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00/ipa-full.tar
    Backed up to /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00
    The ipa-backup command was successful

Verification Steps

  • Ensure that the backup directory contains an encrypted archive with a .gpg file extension.

    [root@server ~]# ls /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00
    header  ipa-full.tar.gpg

Additional resources

8.6. Creating a GPG2 key

The following procedure describes how to generate a GPG2 key to use with encryption utilities.

Prerequisites

  • You need root privileges.

Procedure

  1. Install and configure the pinentry utility.

    [root@server ~]# yum install pinentry
    [root@server ~]# mkdir ~/.gnupg -m 700
    [root@server ~]# echo "pinentry-program /usr/bin/pinentry-curses" >> ~/.gnupg/gpg-agent.conf
  2. Create a key-input file used for generating a GPG keypair with your preferred details. For example:

    [root@server ~]# cat >key-input <<EOF
    %echo Generating a standard key
    Key-Type: RSA
    Key-Length: 2048
    Name-Real: GPG User
    Name-Comment: first key
    Name-Email: root@example.com
    Expire-Date: 0
    %commit
    %echo Finished creating standard key
    EOF
  3. (Optional) By default, GPG2 stores its keyring in the ~/.gnupg file. To use a custom keyring location, set the GNUPGHOME environment variable to a directory that is only accessible by root.

    [root@server ~]# export GNUPGHOME=/root/backup
    
    [root@server ~]# mkdir -p $GNUPGHOME -m 700
  4. Generate a new GPG2 key based on the contents of the key-input file.

    [root@server ~]# gpg2 --batch --gen-key key-input
  5. Enter a passphrase to protect the GPG2 key. You use this passphrase to access the private key for decryption.

    ┌──────────────────────────────────────────────────────┐
    │ Please enter the passphrase to                       │
    │ protect your new key                                 │
    │                                                      │
    │ Passphrase: <passphrase>                             │
    │                                                      │
    │	 <OK>                             <Cancel>         │
    └──────────────────────────────────────────────────────┘
  6. Confirm the correct passphrase by entering it again.

    ┌──────────────────────────────────────────────────────┐
    │ Please re-enter this passphrase                      │
    │                                                      │
    │ Passphrase: <passphrase>                             │
    │                                                      │
    │	 <OK>                             <Cancel>         │
    └──────────────────────────────────────────────────────┘
  7. Verify that the new GPG2 key was created successfully.

    gpg: keybox '/root/backup/pubring.kbx' created
    gpg: Generating a standard key
    gpg: /root/backup/trustdb.gpg: trustdb created
    gpg: key BF28FFA302EF4557 marked as ultimately trusted
    gpg: directory '/root/backup/openpgp-revocs.d' created
    gpg: revocation certificate stored as '/root/backup/openpgp-revocs.d/8F6FCF10C80359D5A05AED67BF28FFA302EF4557.rev'
    gpg: Finished creating standard key

Verification Steps

  • List the GPG keys on the server.

    [root@server ~]# gpg2 --list-secret-keys
    gpg: checking the trustdb
    gpg: marginals needed: 3  completes needed: 1  trust model: pgp
    gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
    /root/backup/pubring.kbx
    ------------------------
    sec   rsa2048 2020-01-13 [SCEA]
          8F6FCF10C80359D5A05AED67BF28FFA302EF4557
    uid           [ultimate] GPG User (first key) <root@example.com>

Additional resources

8.7. When to restore from an IdM backup

You can respond to several disaster scenarios by restoring from an IdM backup:

  • Undesirable changes were made to the LDAP content: Entries were modified or deleted, replication carried out those changes throughout the deployment, and you want to revert those changes. Restoring a data-only backup returns the LDAP entries to the previous state without affecting the IdM configuration itself.
  • Total Infrastructure Loss, or loss of all CA instances: If a disaster damages all Certificate Authority replicas, the deployment has lost the ability to rebuild itself by deploying additional servers. In this situation, restore a backup of a CA Replica and build new replicas from it.
  • An upgrade on an isolated server failed: The operating system remains functional, but the IdM data is corrupted, which is why you want to restore the IdM system to a known good state. Red Hat recommends working with Technical Support to diagnose and troubleshoot the issue. If those efforts fail, restore from a full-server backup.

    Important

    The preferred solution for hardware or upgrade failure is to rebuild the lost server from a replica. For more information, see Recovering a single server with replication.

8.8. Considerations when restoring from an IdM backup

If you have a backup created with the ipa-backup utility, you can restore your IdM server or the LDAP content to the state they were in when the backup was performed.

The following are the key considerations while restoring from an IdM backup:

  • You can only restore a backup on a server that matches the configuration of the server where the backup was originally created. The server must have:

    • The same hostname
    • The same IP address
    • The same version of IdM software
  • If one IdM server among many is restored, the restored server becomes the only source of information for IdM. All other servers must be re-initialized from the restored server.
  • Since any data created after the last backup will be lost, do not use the backup and restore solution for normal system maintenance.
  • If a server is lost, Red Hat recommends rebuilding the server by reinstalling it as a replica, instead of restoring from a backup. Creating a new replica preserves data from the current working environment. For more information, see Preparing for server loss with replication.
  • The backup and restore features can only be managed from the command line and are not available in the IdM web UI.
  • You cannot restore from backup files located in the /tmp or /var/tmp directories. The IdM Directory Server uses a PrivateTmp directory and cannot access the /tmp or /var/tmp directories commonly available to the operating system.
Tip

Restoring from a backup requires the same software (RPM) versions on the target host as were installed when the backup was performed. Due to this, Red Hat recommends restoring from a Virtual Machine snapshot rather than a backup. For more information, see Recovering from data loss with VM snapshots.

8.9. Restoring an IdM server from a backup

The following procedure describes restoring an IdM server, or its LDAP data, from an IdM backup.

Figure 8.1. Replication Topology used in this example

Diagram showing 3 IdM servers: host server1.example.com needs to be restored from backup. Host caReplica2.example.com is a Certificate Authority replica that is connected to server1.example.com. Host replica3.example.com is a regular server that is connected to caReplica2.example.com

Table 8.1. Server naming conventions used in this example

Server host nameFunction

server1.example.com

The server that needs to be restored from backup.

caReplica2.example.com

A Certificate Authority (CA) replica connected to the server1.example.com host.

replica3.example.com

A replica connected to the caReplica2.example.com host.

Prerequisites

  • You have generated a full-server or data-only backup of the IdM server with the ipa-backup utility. See Creating a backup.
  • Your backup files are not in the /tmp or /var/tmp directories.
  • Before performing a full-server restore from a full-server backup, uninstall IdM from the server and reinstall IdM using the same server configuration as before.

Procedure

  1. Use the ipa-restore utility to restore a full-server or data-only backup.

    • If the backup directory is in the default /var/lib/ipa/backup/ location, enter only the name of the directory:

      [root@server1 ~]# ipa-restore ipa-full-2020-01-14-12-02-32
    • If the backup directory is not in the default location, enter its full path:

      [root@server1 ~]# ipa-restore /mybackups/ipa-data-2020-02-01-05-30-00
      Note

      The ipa-restore utility automatically detects the type of backup that the directory contains, and performs the same type of restore by default. To perform a data-only restore from a full-server backup, add the --data option to the ipa-restore command:

      [root@server1 ~]# ipa-restore --data ipa-full-2020-01-14-12-02-32
  2. Enter the Directory Manager password.

    Directory Manager (existing master) password:
  3. Enter yes to confirm overwriting current data with the backup.

    Preparing restore from /var/lib/ipa/backup/ipa-full-2020-01-14-12-02-32 on server1.example.com
    Performing FULL restore from FULL backup
    Temporary setting umask to 022
    Restoring data will overwrite existing live data. Continue to restore? [no]: yes
  4. The ipa-restore utility disables replication on all servers that are available:

    Each master will individually need to be re-initialized or
    re-created from this one. The replication agreements on
    masters running IPA 3.1 or earlier will need to be manually
    re-enabled. See the man page for details.
    Disabling all replication.
    Disabling replication agreement on server1.example.com to caReplica2.example.com
    Disabling CA replication agreement on server1.example.com to caReplica2.example.com
    Disabling replication agreement on caReplica2.example.com to server1.example.com
    Disabling replication agreement on caReplica2.example.com to replica3.example.com
    Disabling CA replication agreement on caReplica2.example.com to server1.example.com
    Disabling replication agreement on replica3.example.com to caReplica2.example.com

    The utility then stops IdM services, restores the backup, and restarts the services:

    Stopping IPA services
    Systemwide CA database updated.
    Restoring files
    Systemwide CA database updated.
    Restoring from userRoot in EXAMPLE-COM
    Restoring from ipaca in EXAMPLE-COM
    Restarting GSS-proxy
    Starting IPA services
    Restarting SSSD
    Restarting oddjobd
    Restoring umask to 18
    The ipa-restore command was successful
  5. Re-initialize all replicas connected to the restored server:

    1. List all replication topology segments for the domain suffix, taking note of topology segments involving the restored server.

      [root@server1 ~]# ipa topologysegment-find domain
      ------------------
      2 segments matched
      ------------------
        Segment name: server1.example.com-to-caReplica2.example.com
        Left node: server1.example.com
        Right node: caReplica2.example.com
        Connectivity: both
      
        Segment name: caReplica2.example.com-to-replica3.example.com
        Left node: caReplica2.example.com
        Right node: replica3.example.com
        Connectivity: both
      ----------------------------
      Number of entries returned 2
      ----------------------------
    2. Re-initialize the domain suffix for all topology segments with the restored server.

      In this example, perform a re-initialization of caReplica2 with data from server1.

      [root@caReplica2 ~]# ipa-replica-manage re-initialize --from=server1.example.com
      Update in progress, 2 seconds elapsed
      Update succeeded
    3. Moving on to Certificate Authority data, list all replication topology segments for the ca suffix.

      [root@server1 ~]# ipa topologysegment-find ca
      -----------------
      1 segment matched
      -----------------
        Segment name: server1.example.com-to-caReplica2.example.com
        Left node: server1.example.com
        Right node: caReplica2.example.com
        Connectivity: both
      ----------------------------
      Number of entries returned 1
      ----------------------------
    4. Re-initialize all CA replicas connected to the restored server.

      In this example, perform a csreplica re-initialization of caReplica2 with data from server1.

      [root@caReplica2 ~]# ipa-csreplica-manage re-initialize --from=server1.example.com
      Directory Manager password:
      
      Update in progress, 3 seconds elapsed
      Update succeeded
  6. Continue moving outward through the replication topology, re-initializing successive replicas, until all servers have been updated with the data from restored server server1.example.com.

    In this example, we only have to re-initialize the domain suffix on replica3 with the data from caReplica2:

    [root@replica3 ~]# ipa-replica-manage re-initialize --from=caReplica2.example.com
    Directory Manager password:
    
    Update in progress, 3 seconds elapsed
    Update succeeded
  7. Clear SSSD’s cache on every server to avoid authentication problems due to invalid data:

    1. Stop the SSSD service:

      [root@server ~]# systemctl stop sssd
    2. Remove all cached content from SSSD:

      [root@server ~]# sss_cache -E
    3. Start the SSSD service:

      [root@server ~]# systemctl start sssd
    4. Reboot the server.

Additional resources

  • The ipa-restore (1) man page also covers in detail how to handle complex replication scenarios during restoration.

8.10. Restoring from an encrypted backup

This procedure restores an IdM server from an encrypted IdM backup. The ipa-restore utility automatically detects if an IdM backup is encrypted and restores it using the GPG2 root keyring.

Prerequisites

Procedure

  1. If you used a custom keyring location when creating the GPG2 keys, verify that the $GNUPGHOME environment variable is set to that directory. See Creating a GPG2 key.

    [root@server ~]# echo $GNUPGHOME
    /root/backup
  2. Provide the ipa-restore utility with the backup directory location.

    [root@server ~]# ipa-restore ipa-full-2020-01-13-18-30-54
    1. Enter the Directory Manager password.

      Directory Manager (existing master) password:
    2. Enter the passphrase you used when creating the GPG key.

      ┌────────────────────────────────────────────────────────────────┐
      │ Please enter the passphrase to unlock the OpenPGP secret key:  │
      │ "GPG User (first key) <root@example.com>"                      │
      │ 2048-bit RSA key, ID BF28FFA302EF4557,                         │
      │ created 2020-01-13.                                            │
      │                                                                │
      │                                                                │
      │ Passphrase: <passphrase>                                       │
      │                                                                │
      │         <OK>                                    <Cancel>       │
      └────────────────────────────────────────────────────────────────┘
  3. Re-initialize all replicas connected to the restored server. See Restoring an IdM server from backup.