Chapter 4. Recovering from data loss with IdM backups

You can use the ipa-restore utility to restore an IdM server to a previous state captured in an IdM backup.

4.1. When to restore from an IdM backup

You can respond to several disaster scenarios by restoring from an IdM backup:

  • Undesirable changes were made to the LDAP content: Entries were modified or deleted, replication carried out those changes throughout the deployment, and you want to revert those changes. Restoring a data-only backup returns the LDAP entries to the previous state without affecting the IdM configuration itself.
  • Total Infrastructure Loss, or loss of all CA instances: If a disaster damages all Certificate Authority replicas, the deployment has lost the ability to rebuild itself by deploying additional servers. In this situation, restore a backup of a CA Replica and build new replicas from it.
  • An upgrade on an isolated server failed: The operating system remains functional, but the IdM data is corrupted, which is why you want to restore the IdM system to a known good state. Red Hat recommends working with Technical Support in order to diagnose and troubleshoot the issue. If those efforts fail, restore from a full-server backup.

    Important

    The preferred solution for hardware or upgrade failure is to rebuild the lost server from a replica. For more information, see Recovering from server loss with replication.

4.2. Considerations when restoring from an IdM backup

If you have a backup created with the ipa-backup utility, you can restore your IdM server or the LDAP content to the state they were in when the backup was performed.

The following are the key considerations while restoring from an IdM backup:

  • You can only restore a backup on a server that matches the configuration of the server where the backup was originally created. The server must have:

    • The same hostname
    • The same IP address
    • The same version of IdM software
  • If one IdM server among many is restored, the restored server becomes the only source of information for IdM. All other servers must be re-initialized from the restored server.
  • Since any data created after the last backup will be lost, do not use the backup and restore solution for normal system maintenance.
  • If a server is lost, Red Hat recommends rebuilding the server by reinstalling it as a replica, instead of restoring from a backup. Creating a new replica preserves data from the current working environment. For more information, see Preparing for server loss with replication.
  • The backup and restore features can only be managed from the command line and are not available in the IdM web UI.
  • You cannot restore from backup files located in the /tmp or /var/tmp directories. The IdM Directory Server uses a PrivateTmp directory and cannot access the /tmp or /var/tmp directories commonly available to the operating system.
Tip

Restoring from a backup requires the same software (RPM) versions on the target host as were installed when the backup was performed. Due to this, Red Hat recommends restoring from a Virtual Machine snapshot rather than a backup. For more information, see Recovering from data loss with VM snapshots.

4.3. Restoring an IdM server from a backup

The following procedure describes restoring an IdM server, or its LDAP data, from an IdM backup.

Figure 4.1. Replication Topology used in this example

Table 4.1. Server naming conventions used in this example

Server host nameFunction

server1.example.com

The server that needs to be restored from backup.

caReplica2.example.com

A Certificate Authority (CA) replica connected to the server1.example.com host.

replica3.example.com

A replica connected to the caReplica2.example.com host.

Prerequisites

  • You have generated a full-server or data-only backup of the IdM server with the ipa-backup utility. See Creating a backup.
  • Your backup files are not in the /tmp or /var/tmp directories.
  • Before performing a full-server restore from a full-server backup, uninstall IdM from the server and reinstall IdM using the same server configuration as before.

Procedure

  1. Use the ipa-restore utility to restore a full-server or data-only backup.

    • If the backup directory is in the default /var/lib/ipa/backup/ location, enter only the name of the directory:

      [root@server1 ~]# ipa-restore ipa-full-2020-01-14-12-02-32
    • If the backup directory is not in the default location, enter its full path:

      [root@server1 ~]# ipa-restore /mybackups/ipa-data-2020-02-01-05-30-00
      Note

      The ipa-restore utility automatically detects the type of backup that the directory contains, and performs the same type of restore by default. To perform a data-only restore from a full-server backup, add the --data option to the ipa-restore command:

      [root@server1 ~]# ipa-restore --data ipa-full-2020-01-14-12-02-32
  2. Enter the Directory Manager password.

    Directory Manager (existing master) password:
  3. Enter yes to confirm overwriting current data with the backup.

    Preparing restore from /var/lib/ipa/backup/ipa-full-2020-01-14-12-02-32 on server1.example.com
    Performing FULL restore from FULL backup
    Temporary setting umask to 022
    Restoring data will overwrite existing live data. Continue to restore? [no]: yes
  4. The ipa-restore utility disables replication on all servers that are available:

    Each master will individually need to be re-initialized or
    re-created from this one. The replication agreements on
    masters running IPA 3.1 or earlier will need to be manually
    re-enabled. See the man page for details.
    Disabling all replication.
    Disabling replication agreement on server1.example.com to caReplica2.example.com
    Disabling CA replication agreement on server1.example.com to caReplica2.example.com
    Disabling replication agreement on caReplica2.example.com to server1.example.com
    Disabling replication agreement on caReplica2.example.com to replica3.example.com
    Disabling CA replication agreement on caReplica2.example.com to server1.example.com
    Disabling replication agreement on replica3.example.com to caReplica2.example.com

    The utility then stops IdM services, restores the backup, and restarts the services:

    Stopping IPA services
    Systemwide CA database updated.
    Restoring files
    Systemwide CA database updated.
    Restoring from userRoot in EXAMPLE-COM
    Restoring from ipaca in EXAMPLE-COM
    Restarting GSS-proxy
    Starting IPA services
    Restarting SSSD
    Restarting oddjobd
    Restoring umask to 18
    The ipa-restore command was successful
  5. Re-initialize all replicas connected to the restored server:

    1. List all replication topology segments for the domain suffix, taking note of topology segments involving the restored server.

      [root@server1 ~]# ipa topologysegment-find domain
      ------------------
      2 segments matched
      ------------------
        Segment name: server1.example.com-to-caReplica2.example.com
        Left node: server1.example.com
        Right node: caReplica2.example.com
        Connectivity: both
      
        Segment name: caReplica2.example.com-to-replica3.example.com
        Left node: caReplica2.example.com
        Right node: replica3.example.com
        Connectivity: both
      ----------------------------
      Number of entries returned 2
      ----------------------------
    2. Re-initialize the domain suffix for all topology segments with the restored server.

      In this example, perform a re-initialization of caReplica2 with data from server1.

      [root@caReplica2 ~]# ipa-replica-manage re-initialize --from=server1.example.com
      Update in progress, 2 seconds elapsed
      Update succeeded
    3. Moving on to Certificate Authority data, list all replication topology segments for the ca suffix.

      [root@server1 ~]# ipa topologysegment-find ca
      -----------------
      1 segment matched
      -----------------
        Segment name: server1.example.com-to-caReplica2.example.com
        Left node: server1.example.com
        Right node: caReplica2.example.com
        Connectivity: both
      ----------------------------
      Number of entries returned 1
      ----------------------------
    4. Re-initialize all CA replicas connected to the restored server.

      In this example, perform a csreplica re-initialization of caReplica2 with data from server1.

      [root@caReplica2 ~]# ipa-csreplica-manage re-initialize --from=server1.example.com
      Directory Manager password:
      
      Update in progress, 3 seconds elapsed
      Update succeeded
  6. Continue moving outward through the replication topology, re-initializing successive replicas, until all servers have been updated with the data from restored server server1.example.com.

    In this example, we only have to re-initialize the domain suffix on replica3 with the data from caReplica2:

    [root@replica3 ~]# ipa-replica-manage re-initialize --from=caReplica2.example.com
    Directory Manager password:
    
    Update in progress, 3 seconds elapsed
    Update succeeded
  7. Clear SSSD’s cache on every server in order to avoid authentication problems due to invalid data:

    1. Stop the SSSD service:

      [root@server ~]# systemctl stop sssd
    2. Remove all cached content from SSSD:

      [root@server ~]# sss_cache -E
    3. Start the SSSD service:

      [root@server ~]# systemctl start sssd
    4. Reboot the server.

Additional resources

  • The ipa-restore (1) man page also covers in detail how to handle complex replication scenarios during restoration.

4.4. Restoring from an encrypted backup

This procedure restores an IdM server from an encrypted IdM backup. The ipa-restore utility automatically detects if an IdM backup is encrypted and restores it using the GPG2 root keyring.

Prerequisites

Procedure

  1. If you used a custom keyring location when creating the GPG2 keys, make sure that the $GNUPGHOME environment variable is set to that directory. See Creating a GPG2 key.

    [root@server ~]# echo $GNUPGHOME
    /root/backup
  2. Provide the ipa-restore utility with the backup directory location.

    [root@server ~]# ipa-restore ipa-full-2020-01-13-18-30-54
    1. Enter the Directory Manager password.

      Directory Manager (existing master) password:
    2. Enter the passphrase you used when creating the GPG key.

      ┌────────────────────────────────────────────────────────────────┐
      │ Please enter the passphrase to unlock the OpenPGP secret key:  │
      │ "GPG User (first key) <root@example.com>"                      │
      │ 2048-bit RSA key, ID BF28FFA302EF4557,                         │
      │ created 2020-01-13.                                            │
      │                                                                │
      │                                                                │
      │ Passphrase: <passphrase>                                       │
      │                                                                │
      │         <OK>                                    <Cancel>       │
      └────────────────────────────────────────────────────────────────┘
  3. Re-initialize all replicas connected to the restored server. See Restoring an IdM server from backup.

4.5. Restoring IdM servers using Ansible playbooks

Using the ipabackup Ansible role, you can automate restoring an IdM server from a backup and transferring backup files between servers and your Ansible controller.

This section covers the following topics:

4.5.1. Preparing your Ansible control node for managing IdM

When working with Ansible as a system administrator managing Identity Management (IdM), it is good practice to create a subdirectory dedicated to Ansible playbooks in your home directory, for example ~/MyPlaybooks. In order to use Ansible for your purposes, copy and adapt sample Ansible playbooks from the /usr/share/doc/ansible-freeipa/* and /usr/share/doc/rhel-system-roles/* directories and subdirectories into the ~/MyPlaybooks directory. This practice has the following advantages:

  • You can find all your playbooks in one place.
  • You can run your playbooks without invoking root privileges.

It is good practice to also include your inventory file in the ~/MyPlaybooks/ directory.

This section describes how to create the ~/MyPlaybooks/ directory and configure it so that you can use it to store and run Ansible playbooks.

Prerequisites

  • You have installed an IdM server on your managed nodes, server.idm.example.com and replica.idm.example.com.
  • You have configured DNS and networking so you can log in to the managed nodes, server.idm.example.com and replica.idm.example.com, directly from the control node.
  • You know the IdM admin password.

Procedure

  1. Create a directory for your Ansible configuration and playbooks in your home directory:

    $ mkdir ~/MyPlaybooks/
  2. Change into the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks
  3. Create the ~/MyPlaybooks/ansible.cfg file with the following content:

    [defaults]
    inventory = /home/your_username/MyPlaybooks/inventory
    
    [privilege_escalation]
    become=True
  4. Create the ~/MyPlaybooks/inventory file with the following content:

    [eu]
    server.idm.example.com
    
    [us]
    replica.idm.example.com
    
    [ipaserver:children]
    eu
    us

    This configuration defines two host groups, eu and us, for hosts in these locations. Additionally, this configuration defines the ipaserver host group, which contains all hosts from the eu and us groups.

  5. [Optional] Create an SSH public and private key. To simplify access in your test environment, do not set a password on the private key:

    $ ssh-keygen
  6. Copy the SSH public key to the IdM admin account on each managed node:

    $ ssh-copy-id admin@server.idm.example.com
    $ ssh-copy-id admin@replica.idm.example.com

    These commands require that you enter the IdM admin password.

Additional resources

4.5.2. Using Ansible to restore an IdM server from a backup stored on the server

The following procedure describes how to use an Ansible playbook to restore an IdM server from a backup stored on that host.

Prerequisites

  • You have configured an Ansible control node that meets the following requirements:

    • You are using Ansible version 2.8 or later.
    • You have installed the ansible-freeipa package.
    • You have created an Ansible inventory file with the fully-qualified domain name (FQDN) of the IdM server where you are configuring these options.
    • Your Ansible inventory file is located in the ~/MyPlaybooks/ directory.
  • You know the LDAP Directory Manager password.

Procedure

  1. Navigate to the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks/
  2. Make a copy of the restore-server.yml file located in the /usr/share/doc/ansible-freeipa/playbooks directory:

    $ cp /usr/share/doc/ansible-freeipa/playbooks/restore-server.yml restore-my-server.yml
  3. Open the restore-my-server.yml Ansible playbook file for editing.
  4. Adapt the file by setting the following variables:

    1. Set the hosts variable to a host group from your inventory file. In this example, set it to the ipaserver host group.
    2. Set the ipabackup_name variable to the name of the ipabackup to restore.
    3. Set the ipabackup_password variable to the LDAP Directory Manager password.

      ---
      - name: Playbook to restore an IPA server
        hosts: ipaserver
        become: true
      
        vars:
          ipabackup_name: ipa-full-2021-04-30-13-12-00
          ipabackup_password: <your_LDAP_DM_password>
      
        roles:
        - role: ipabackup
          state: restored
  5. Save the file.
  6. Run the Ansible playbook specifying the inventory file and the playbook file:

    $ ansible-playbook -v -i ~/MyPlaybooks/inventory restore-my-server.yml

Additional resources

  • For more sample Ansible playbooks that use the ipabackup role, see:

    • The README.md file in the /usr/share/doc/ansible-freeipa/roles/ipabackup directory. This file also contains the definitions of the ipabackup variables.
    • The /usr/share/doc/ansible-freeipa/playbooks/ directory.

4.5.3. Using Ansible to restore an IdM server from a backup stored on your Ansible controller

The following procedure describes how to use an Ansible playbook to restore an IdM server from a backup stored on your Ansible controller.

Prerequisites

  • You have configured an Ansible control node that meets the following requirements:

    • You are using Ansible version 2.8 or later.
    • You have installed the ansible-freeipa package.
    • You have created an Ansible inventory file with the fully-qualified domain name (FQDN) of the IdM server where you are configuring these options.
    • Your Ansible inventory file is located in the ~/MyPlaybooks/ directory.
  • You know the LDAP Directory Manager password.

Procedure

  1. Navigate to the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks/
  2. Make a copy of the restore-server-from-controller.yml file located in the /usr/share/doc/ansible-freeipa/playbooks directory:

    $ cp /usr/share/doc/ansible-freeipa/playbooks/restore-server-from-controller.yml restore-my-server-from-my-controller.yml
  3. Open the restore-my-server-from-my-controller.yml file for editing.
  4. Adapt the file by setting the following variables:

    1. Set the hosts variable to a host group from your inventory file. In this example, set it to the ipaserver host group.
    2. Set the ipabackup_name variable to the name of the ipabackup to restore.
    3. Set the ipabackup_password variable to the LDAP Directory Manager password.

      ---
      - name: Playbook to restore IPA server from controller
        hosts: ipaserver
        become: true
      
        vars:
          ipabackup_name: server.idm.example.com_ipa-full-2021-04-30-13-12-00
          ipabackup_password: <your_LDAP_DM_password>
          ipabackup_from_controller: yes
      
        roles:
        - role: ipabackup
          state: restored
  5. Save the file.
  6. Run the Ansible playbook, specifying the inventory file and the playbook file:

    $ ansible-playbook -v -i ~/MyPlaybooks/inventory restore-my-server-from-my-controller.yml

Additional resources

  • For more sample Ansible playbooks that use the ipabackup role, see:

    • The README.md file in the /usr/share/doc/ansible-freeipa/roles/ipabackup directory. This file also contains the definitions of the ipabackup variables.
    • The /usr/share/doc/ansible-freeipa/playbooks/ directory.

4.5.4. Using Ansible to copy a backup of an IdM server to your Ansible controller

The following procedure describes how to use an Ansible playbook to copy a backup of an IdM server from the IdM server to your Ansible controller.

Prerequisites

  • You have configured an Ansible control node that meets the following requirements:

    • You are using Ansible version 2.8 or later.
    • You have installed the ansible-freeipa package.
    • You have created an Ansible inventory file with the fully-qualified domain name (FQDN) of the IdM server where you are configuring these options.
    • Your Ansible inventory file is located in the ~/MyPlaybooks/ directory.

Procedure

  1. To store the backups, create a subdirectory in your home directory on the Ansible controller.

    $ mkdir ~/ipabackups
  2. Navigate to the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks/
  3. Make a copy of the copy-backup-from-server.yml file located in the /usr/share/doc/ansible-freeipa/playbooks directory:

    $ cp /usr/share/doc/ansible-freeipa/playbooks/copy-backup-from-server.yml copy-backup-from-my-server-to-my-controller.yml
  4. Open the copy-my-backup-from-my-server-to-my-controller.yml file for editing.
  5. Adapt the file by setting the following variables:

    1. Set the hosts variable to a host group from your inventory file. In this example, set it to the ipaserver host group.
    2. Set the ipabackup_name variable to the name of the ipabackup on your IdM server to copy to your Ansible controller.
    3. By default, backups are stored in the present working directory of the Ansible controller. To specify the directory you created in Step 1, add the ipabackup_controller_path variable and set it to the /home/user/ipabackups directory.

      ---
      - name: Playbook to copy backup from IPA server
        hosts: ipaserver
        become: true
        vars:
          ipabackup_name: ipa-full-2021-04-30-13-12-00
          ipabackup_to_controller: yes
          ipabackup_controller_path: /home/user/ipabackups
      
        roles:
        - role: ipabackup
          state: present
  6. Save the file.
  7. Run the Ansible playbook, specifying the inventory file and the playbook file:

    $ ansible-playbook -v -i ~/MyPlaybooks/inventory copy-backup-from-my-server-to-my-controller.yml
Note

To copy all IdM backups to your controller, set the ipabackup_name variable in the Ansible playbook to all:

  vars:
    ipabackup_name: all
    ipabackup_to_controller: yes

For an example, see the copy-all-backups-from-server.yml Ansible playbook in the /usr/share/doc/ansible-freeipa/playbooks directory.

Verification steps

  • Verify your backup is in the /home/user/ipabackups directory on your Ansible controller:

    [user@controller ~]$ ls /home/user/ipabackups
    server.idm.example.com_ipa-full-2021-04-30-13-12-00

Additional resources

  • For more sample Ansible playbooks that use the ipabackup role, see:

    • The README.md file in the /usr/share/doc/ansible-freeipa/roles/ipabackup directory. This file also contains the definitions of the ipabackup variables.
    • The /usr/share/doc/ansible-freeipa/playbooks/ directory.

4.5.5. Using Ansible to copy a backup of an IdM server from your Ansible controller to the IdM server

The following procedure describes how to use an Ansible playbook to copy a backup of an IdM server from your Ansible controller to the IdM server.

Prerequisites

  • You have configured an Ansible control node that meets the following requirements:

    • You are using Ansible version 2.8 or later.
    • You have installed the ansible-freeipa package.
    • You have created an Ansible inventory file with the fully-qualified domain name (FQDN) of the IdM server where you are configuring these options.
    • Your Ansible inventory file is located in the ~/MyPlaybooks/ directory.

Procedure

  1. Navigate to the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks/
  2. Make a copy of the copy-backup-from-controller.yml file located in the /usr/share/doc/ansible-freeipa/playbooks directory:

    $ cp /usr/share/doc/ansible-freeipa/playbooks/copy-backup-from-controller.yml copy-backup-from-my-controller-to-my-server.yml
  3. Open the copy-my-backup-from-my-controller-to-my-server.yml file for editing.
  4. Adapt the file by setting the following variables:

    1. Set the hosts variable to a host group from your inventory file. In this example, set it to the ipaserver host group.
    2. Set the ipabackup_name variable to the name of the ipabackup on your Ansible controller to copy to the IdM server.

      ---
      - name: Playbook to copy a backup from controller to the IPA server
        hosts: ipaserver
        become: true
      
        vars:
          ipabackup_name: server.idm.example.com_ipa-full-2021-04-30-13-12-00
          ipabackup_from_controller: yes
      
        roles:
        - role: ipabackup
          state: copied
  5. Save the file.
  6. Run the Ansible playbook, specifying the inventory file and the playbook file:

    $ ansible-playbook -v -i ~/MyPlaybooks/inventory copy-backup-from-my-controller-to-my-server.yml

Additional resources

  • For more sample Ansible playbooks that use the ipabackup role, see:

    • The README.md file in the /usr/share/doc/ansible-freeipa/roles/ipabackup directory. This file also contains the definitions of the ipabackup variables.
    • The /usr/share/doc/ansible-freeipa/playbooks/ directory.

4.5.6. Using Ansible to remove a backup from an IdM server

The following procedure describes how to use an Ansible playbook to remove a backup from an IdM server.

Prerequisites

  • You have configured an Ansible control node that meets the following requirements:

    • You are using Ansible version 2.8 or later.
    • You have installed the ansible-freeipa package.
    • You have created an Ansible inventory file with the fully-qualified domain name (FQDN) of the IdM server where you are configuring these options.
    • Your Ansible inventory file is located in the ~/MyPlaybooks/ directory.

Procedure

  1. Navigate to the ~/MyPlaybooks/ directory:

    $ cd ~/MyPlaybooks/
  2. Make a copy of the remove-backup-from-server.yml file located in the /usr/share/doc/ansible-freeipa/playbooks directory:

    $ cp /usr/share/doc/ansible-freeipa/playbooks/remove-backup-from-server.yml remove-backup-from-my-server.yml
  3. Open the remove-backup-from-my-server.yml file for editing.
  4. Adapt the file by setting the following variables:

    1. Set the hosts variable to a host group from your inventory file. In this example, set it to the ipaserver host group.
    2. Set the ipabackup_name variable to the name of the ipabackup to remove from your IdM server.

      ---
      - name: Playbook to remove backup from IPA server
        hosts: ipaserver
        become: true
      
        vars:
          ipabackup_name: ipa-full-2021-04-30-13-12-00
      
        roles:
        - role: ipabackup
          state: absent
  5. Save the file.
  6. Run the Ansible playbook, specifying the inventory file and the playbook file:

    $ ansible-playbook -v -i ~/MyPlaybooks/inventory remove-backup-from-my-server.yml
Note

To remove all IdM backups from the IdM server, set the ipabackup_name variable in the Ansible playbook to all:

  vars:
    ipabackup_name: all

For an example, see the remove-all-backups-from-server.yml Ansible playbook in the /usr/share/doc/ansible-freeipa/playbooks directory.

Additional resources

  • For more sample Ansible playbooks that use the ipabackup role, see:

    • The README.md file in the /usr/share/doc/ansible-freeipa/roles/ipabackup directory. This file also contains the definitions of the ipabackup variables.
    • The /usr/share/doc/ansible-freeipa/playbooks/ directory.