Preparing for disaster recovery with Identity Management
Documentation for mitigating disasters affecting an Identity Management deployment
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your input on our documentation. Please let us know how we could make it better. To do so:
For simple comments on specific passages:
- Make sure you are viewing the documentation in the Multi-page HTML format. In addition, ensure you see the Feedback button in the upper right corner of the document.
- Use your mouse cursor to highlight the part of text that you want to comment on.
- Click the Add Feedback pop-up that appears below the highlighted text.
- Follow the displayed instructions.
For submitting more complex feedback, create a Bugzilla ticket:
- Go to the Bugzilla website.
- As the Component, use Documentation.
- Fill in the Description field with your suggestion for improvement. Include a link to the relevant part(s) of documentation.
- Click Submit Bug.
Chapter 1. Disaster recovery tools in IdM
A good disaster recovery strategy combines the following tools in order to recover from a disaster as soon as possible with minimal data loss:
- Replication copies database contents between IdM servers. If an IdM server fails, you can replace the lost server by creating a new replica based on one of the remaining servers.
- Virtual machine (VM) snapshots
- A snapshot is a view of a VM’s operating system and applications on any or all available disks at a given point in time. After taking a VM snapshot, you can use it to return a VM and its IdM data to a previous state.
- IdM backups
ipa-backuputility allows you to take a backup of an IdM server’s configuration files and its data. You can later use a backup to restore an IdM server to a previous state.
Chapter 2. Disaster scenarios in IdM
There are two main classes of disaster scenarios: server loss and data loss.
Table 2.1. Server loss vs. data loss
How to prepare
Server loss: The IdM deployment loses one or several servers.
Data loss: IdM data is unexpectedly modified on a server, and the change is propagated to other servers.
Chapter 3. Preparing for server loss with replication
Follow these guidelines to establish a replication topology that will allow you to respond to losing a server.
3.1. Connecting the replicas in a topology
- Connect each replica to at least two other replicas
- Configuring additional replication agreements ensures that information is replicated not just between the initial replica and the first server you installed, but between other replicas as well.
- Connect a replica to a maximum of four other replicas (not a hard requirement)
A large number of replication agreements per server does not add significant benefits. A receiving replica can only be updated by one other replica at a time and meanwhile, the other replication agreements are idle. More than four replication agreements per replica typically means a waste of resources.Note
This recommendation applies to both certificate replication and domain replication agreements.
There are two exceptions to the limit of four replication agreements per replica:
- You want failover paths if certain replicas are not online or responding.
- In larger deployments, you want additional direct links between specific nodes.
Configuring a high number of replication agreements can have a negative impact on overall performance: when multiple replication agreements in the topology are sending updates, certain replicas can experience a high contention on the changelog database file between incoming updates and the outgoing updates.
If you decide to use more replication agreements per replica, ensure that you do not experience replication issues and latency. However, note that large distances and high numbers of intermediate nodes can also cause latency problems.
- Connect the replicas in a data center with each other
- This ensures domain replication within the data center.
- Connect each data center to at least two other data centers
- This ensures domain replication between data centers.
- Connect data centers using at least a pair of replication agreements
- If data centers A and B have a replication agreement from A1 to B1, having a replication agreement from A2 to B2 ensures that if one of the servers is down, the replication can continue between the two data centers.
3.2. Replica topology examples
The figures below show examples of Identity Management (IdM) topologies based on the guidelines for creating a reliable topology.
Figure 3.1, “Replica Topology Example 1” shows four data centers, each with four servers. The servers are connected with replication agreements.
Figure 3.1. Replica Topology Example 1
Figure 3.2, “Replica Topology Example 2” shows three data centers, each with a different number of servers. The servers are connected with replication agreements.
Figure 3.2. Replica Topology Example 2
3.3. Protecting IdM CA data
If your deployment contains the integrated IdM Certificate Authority (CA), install several CA replicas so you can create additional CA replicas if one is lost.
Configure three or more replicas to provide CA services.
To install a new replica with CA services, run
[root@server ~]# ipa-replica-install --setup-ca
To install CA services on a preexisting replica, run
[root@replica ~]# ipa-ca-install
Create CA replication agreements between your CA replicas.
[root@careplica1 ~]# ipa topologysegment-add Suffix name: ca Left node: ca-replica1.example.com Right node: ca-replica2.example.com Segment name [ca-replica1.example.com-to-ca-replica2.example.com]: new_segment --------------------------- Added segment "new_segment" --------------------------- Segment name: new_segment Left node: ca-replica1.example.com Right node: ca-replica2.example.com Connectivity: both
If only one server provides CA services and it is damaged, the entire environment will be lost. If you use the IdM CA, Red Hat strongly recommends having three or more replicas with CA services installed, with CA replication agreements between them.
3.4. Additional resources
- For additional information on replication, see Planning the replica topology.
Chapter 4. Preparing for data loss with VM snapshots
Virtual machine (VM) snapshots are an integral component of a data recovery strategy, since they preserve the full state of an IdM server:
- Operating system software and settings
- IdM software and settings
- IdM customer data
Preparing a VM snapshot of an IdM Certificate Authority (CA) replica allows you to rebuild an entire IdM deployment after a disaster.
If your environment uses the integrated CA, a snapshot of a replica without a CA will not be sufficient for rebuilding a deployment, because certificate data will not be preserved.
Similarly, if your environment uses the IdM Key Recovery Authority (KRA), make sure you create snapshots of a KRA replica, or you may lose the storage key.
Red Hat recommends creating snapshots of a VM that has all of the IdM server roles installed which are in use in your deployment: CA, KRA, DNS.
- A hypervisor capable of hosting RHEL VMs.
Configure at least one CA replica in the deployment to run inside a VM.
- If IdM DNS or KRA are used in your environment, consider installing DNS and KRA services on this replica as well.
- Optionally, configure this VM replica as a hidden replica.
- Periodically shutdown this VM, take a full snapshot of it, and bring it back online so it continues to receive replication updates. If the VM is a hidden replica, IdM Clients will not be disrupted during this procedure.
- For a list of certified hypervisors that have been tested and proven to run Red Hat Enterprise Linux as a guest, see Which hypervisors are certified to run Red Hat Enterprise Linux?.
- For more information on hidden replicas, see The hidden replica mode.
Chapter 5. Preparing for data loss with IdM backups
IdM provides the
ipa-backup utility to backup IdM data, and the
ipa-restore utility to restore servers and data from those backups.
Red Hat recommends running backups as often as necessary on a hidden replica with all server roles installed, especially the Certificate Authority (CA) role if the environment uses the integrated IdM CA. See Installing an IdM hidden replica.
5.1. IdM backup types
ipa-backup utility, you can create two types of backups:
- Full-server backup
- Contains all server configuration files related to IdM, and LDAP data in LDAP Data Interchange Format (LDIF) files
- IdM services must be offline.
- Suitable for rebuilding an IdM deployment from scratch.
- Data-only backup
- Contains LDAP data in LDIF files and the replication changelog
- IdM services can be online or offline.
- Suitable for restoring IdM data to a state in the past
5.2. Naming conventions for IdM backup files
By default, IdM stores backups as
.tar archives in subdirectories of the
The archives and subdirectories follow these naming conventions:
- Full-server backup
An archive named
ipa-full.tarin a directory named
ipa-full-<YEAR-MM-DD-HH-MM-SS>, with the time specified in GMT time.
[root@server ~]# ll /var/lib/ipa/backup/ipa-full-2021-01-29-12-11-46 total 3056 -rw-r--r--. 1 root root 158 Jan 29 12:11 header -rw-r--r--. 1 root root 3121511 Jan 29 12:11 ipa-full.tar
- Data-only backup
An archive named
ipa-data.tarin a directory named
ipa-data-<YEAR-MM-DD-HH-MM-SS>, with the time specified in GMT time.
[root@server ~]# ll /var/lib/ipa/backup/ipa-data-2021-01-29-12-14-23 total 1072 -rw-r--r--. 1 root root 158 Jan 29 12:14 header -rw-r--r--. 1 root root 1090388 Jan 29 12:14 ipa-data.tar
Uninstalling an IdM server does not automatically remove any backup files.
5.3. Considerations when creating a backup
This section describes important behaviors and limitations of the
By default, the
ipa-backuputility runs in offline mode, which stops all IdM services. The utility automatically restarts IdM services after the backup is finished.
- A full-server backup must always run with IdM services offline, but a data-only backup may be performed with services online.
By default, the
ipa-backuputility creates backups on the file system containing the
/var/lib/ipa/backup/directory. Red Hat recommends creating backups regularly on a file system separate from the production filesystem used by IdM, and archiving the backups to a fixed medium, such as tape or optical storage.
- Consider performing backups on hidden replicas. IdM services can be shut down on hidden replicas without affecting IdM clients.
Starting with RHEL 8.3.0, the
ipa-backuputility checks if all of the services used in your IdM cluster, such as a Certificate Authority (CA), Domain Name System (DNS), and Key Recovery Agent (KRA), are installed on the server where you are running the backup. If the server does not have all these services installed, the
ipa-backuputility exits with a warning, because backups taken on that host would not be sufficient for a full cluster restoration.
For example, if your IdM deployment uses an integrated Certificate Authority (CA), a backup run on a non-CA replica will not capture CA data. Red Hat recommends verifying that the replica where you perform an
ipa-backuphas all of the IdM services used in the cluster installed.
You can bypass the IdM server role check with the
ipa-backup --disable-role-checkcommand, but the resulting backup will not contain all the data necessary to restore IdM fully.
5.4. Creating an IdM backup
This section describes how to create a full-server and data-only backup in offline and online modes using the
You must have
rootprivileges to run the
To create a full-server backup in offline mode, use the
ipa-backuputility without additional options.
[root@server ~]# ipa-backup Preparing backup on server.example.com Stopping IPA services Backing up ipaca in EXAMPLE-COM to LDIF Backing up userRoot in EXAMPLE-COM to LDIF Backing up EXAMPLE-COM Backing up files Starting IPA service Backed up to /var/lib/ipa/backup/ipa-full-2020-01-14-11-26-06 The ipa-backup command was successful
To create an offline data-only backup, specify the
[root@server ~]# ipa-backup --data
To create a full-server backup that includes IdM log files, use the
[root@server ~]# ipa-backup --logs
To create a data-only backup while IdM services are running, specify both
[root@server ~]# ipa-backup --data --online
If the backup fails due to insufficient space in the
/tmp directory, use the
TMPDIR environment variable to change the destination for temporary files created by the backup process:
[root@server ~]# TMPDIR=/new/location ipa-backup
For more details, see ipa-backup Command Fails to Finish.
The backup directory contains an archive with the backup.
[root@server ~]# ls /var/lib/ipa/backup/ipa-full-2020-01-14-11-26-06 header ipa-full.tar
5.5. Creating encrypted IdM backups
You can create encrypted backups using GNU Privacy Guard (GPG) encryption. To create encrypted IdM backups, you will first need to create a GPG2 key.
5.5.1. Creating a GPG2 key
The following procedure describes how to generate a GPG2 key to use with encryption utilities.
Install and configure the
[root@server ~]# dnf install pinentry [root@server ~]# mkdir ~/.gnupg -m 700 [root@server ~]# echo "pinentry-program /usr/bin/pinentry-curses" >> ~/.gnupg/gpg-agent.conf
key-inputfile used for generating a GPG keypair with your preferred details. For example:
[root@server ~]# cat >key-input <<EOF %echo Generating a standard key Key-Type: RSA Key-Length: 2048 Name-Real: GPG User Name-Comment: first key Name-Email: firstname.lastname@example.org Expire-Date: 0 %commit %echo Finished creating standard key EOF
(Optional) By default, GPG2 stores its keyring in the
~/.gnupgfile. To use a custom keyring location, set the
GNUPGHOMEenvironment variable to a directory that is only accessible by root.
[root@server ~]# export GNUPGHOME=/root/backup [root@server ~]# mkdir -p $GNUPGHOME -m 700
Generate a new GPG2 key based on the contents of the
[root@server ~]# gpg2 --batch --gen-key key-input
Enter a passphrase to protect the GPG2 key. You use this passphrase to access the private key for decryption.
┌──────────────────────────────────────────────────────┐ │ Please enter the passphrase to │ │ protect your new key │ │ │ │ Passphrase: <passphrase> │ │ │ │ <OK> <Cancel> │ └──────────────────────────────────────────────────────┘
Confirm the correct passphrase by entering it again.
┌──────────────────────────────────────────────────────┐ │ Please re-enter this passphrase │ │ │ │ Passphrase: <passphrase> │ │ │ │ <OK> <Cancel> │ └──────────────────────────────────────────────────────┘
Verify that the new GPG2 key was created successfully.
gpg: keybox '/root/backup/pubring.kbx' created gpg: Generating a standard key gpg: /root/backup/trustdb.gpg: trustdb created gpg: key BF28FFA302EF4557 marked as ultimately trusted gpg: directory '/root/backup/openpgp-revocs.d' created gpg: revocation certificate stored as '/root/backup/openpgp-revocs.d/8F6FCF10C80359D5A05AED67BF28FFA302EF4557.rev' gpg: Finished creating standard key
List the GPG keys on the server.
[root@server ~]# gpg2 --list-secret-keys gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u /root/backup/pubring.kbx ------------------------ sec rsa2048 2020-01-13 [SCEA] 8F6FCF10C80359D5A05AED67BF28FFA302EF4557 uid [ultimate] GPG User (first key) <email@example.com>
- For more information on GPG encryption and its uses, see the GNU Privacy Guard website.
5.5.2. Creating a GPG2-encrypted IdM backup
The following procedure creates an IdM backup and encrypts it using a GPG2 key.
- You have created a GPG2 key. See Creating a GPG2 key.
Create a GPG-encrypted backup by specifying the
[root@server ~]# ipa-backup --gpg Preparing backup on server.example.com Stopping IPA services Backing up ipaca in EXAMPLE-COM to LDIF Backing up userRoot in EXAMPLE-COM to LDIF Backing up EXAMPLE-COM Backing up files Starting IPA service Encrypting /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00/ipa-full.tar Backed up to /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00 The ipa-backup command was successful
Ensure that the backup directory contains an encrypted archive with a
[root@server ~]# ls /var/lib/ipa/backup/ipa-full-2020-01-13-14-38-00 header ipa-full.tar.gpg
- For general information on creating a backup, see Creating a backup.
5.6. Additional resources
- For more information on backing up and restoring IdM, see Backing up and restoring IdM.