Chapter 4. Recovering from data loss with VM snapshots

If a data loss event occurs, you can restore a Virtual Machine (VM) snapshot of a Certificate Authority (CA) replica to repair the lost data, or deploy a new environment from it.

4.1. Recovering from only a VM snapshot

If a disaster affects all IdM servers, and only a snapshot of an IdM CA replica virtual machine (VM) is left, you can recreate your deployment by removing all references to the lost servers and installing new replicas.

Prerequisites

Procedure

  1. Boot the desired snapshot of the CA replica VM.
  2. Remove replication agreements to any lost replicas.

    [root@server ~]# ipa server-del lost-server1.example.com
    [root@server ~]# ipa server-del lost-server2.example.com
    ...
  3. Install a second CA replica. See Installing an IdM replica.
  4. The VM CA replica is now the CA renewal server. Red Hat recommends promoting another CA replica in the environment to act as the CA renewal server. See Changing and resetting IdM CA renewal server.
  5. Recreate the desired replica topology by deploying additional replicas with the desired services (CA, DNS). See Installing an IdM replica
  6. Update DNS to reflect the new replica topology. If IdM DNS is used, DNS service records are updated automatically.
  7. Verify that IdM clients can reach the IdM servers. See Adjusting IdM Clients during recovery.

Verification steps

  1. Test the Kerberos server on every replica by successfully retrieving a Kerberos ticket-granting ticket as an IdM user.

    [root@server ~]# kinit admin
    Password for admin@EXAMPLE.COM:
    
    [root@server ~]# klist
    Ticket cache: KCM:0
    Default principal: admin@EXAMPLE.COM
    
    Valid starting       Expires              Service principal
    10/31/2019 15:51:37  11/01/2019 15:51:02  HTTP/server.example.com@EXAMPLE.COM
    10/31/2019 15:51:08  11/01/2019 15:51:02  krbtgt/EXAMPLE.COM@EXAMPLE.COM
  2. Test the Directory Server and SSSD configuration on every replica by retrieving user information.

    [root@server ~]# ipa user-show admin
      User login: admin
      Last name: Administrator
      Home directory: /home/admin
      Login shell: /bin/bash
      Principal alias: admin@EXAMPLE.COM
      UID: 1965200000
      GID: 1965200000
      Account disabled: False
      Password: True
      Member of groups: admins, trust admins
      Kerberos keys available: True
  3. Test the CA server on every CA replica with the ipa cert-show command.

    [root@server ~]# ipa cert-show 1
      Issuing CA: ipa
      Certificate: MIIEgjCCAuqgAwIBAgIjoSIP...
      Subject: CN=Certificate Authority,O=EXAMPLE.COM
      Issuer: CN=Certificate Authority,O=EXAMPLE.COM
      Not Before: Thu Oct 31 19:43:29 2019 UTC
      Not After: Mon Oct 31 19:43:29 2039 UTC
      Serial number: 1
      Serial number (hex): 0x1
      Revoked: False

Additional resources

4.2. Recovering from a VM snapshot among a partially-working environment

If a disaster affects some IdM servers while others are still operating properly, you may want to restore the deployment to the state captured in a Virtual Machine (VM) snapshot. For example, if all Certificate Authority (CA) Replicas are lost while other replicas are still in production, you will need to bring a CA Replica back into the environment.

In this scenario, remove references to the lost replicas, restore the CA replica from the snapshot, verify replication, and deploy new replicas.

Prerequisites

Procedure

  1. Remove all replication agreements to the lost servers. See Uninstalling an IdM server.
  2. Boot the desired snapshot of the CA replica VM.
  3. Remove any replication agreements between the restored server and any lost servers.

    [root@restored-CA-replica ~]# ipa server-del lost-server1.example.com
    [root@restored-CA-replica ~]# ipa server-del lost-server2.example.com
    ...
  4. If the restored server does not have replication agreements to any of the servers still in production, connect the restored server with one of the other servers to update the restored server.

    [root@restored-CA-replica ~]# ipa topologysegment-add
    Suffix name: domain
    Left node: restored-CA-replica.example.com
    Right node: server3.example.com
    Segment name [restored-CA-replica.com-to-server3.example.com]: new_segment
    ---------------------------
    Added segment "new_segment"
    ---------------------------
      Segment name: new_segment
      Left node: restored-CA-replica.example.com
      Right node: server3.example.com
      Connectivity: both
  5. Review Directory Server error logs at /var/log/dirsrv/slapd-YOUR-INSTANCE/errors to see if the CA replica from the snapshot correctly synchronizes with the remaining IdM servers.
  6. If replication on the restored server fails because its database is too outdated, reinitialize the restored server.

    [root@restored-CA-replica ~]# ipa-replica-manage re-initialize --from server2.example.com
  7. If the database on the restored server is correctly synchronized, continue by deploying additional replicas with the desired services (CA, DNS) according to Installing an IdM replica.

Verification steps

  1. Test the Kerberos server on every replica by successfully retrieving a Kerberos ticket-granting ticket as an IdM user.

    [root@server ~]# kinit admin
    Password for admin@EXAMPLE.COM:
    
    [root@server ~]# klist
    Ticket cache: KCM:0
    Default principal: admin@EXAMPLE.COM
    
    Valid starting       Expires              Service principal
    10/31/2019 15:51:37  11/01/2019 15:51:02  HTTP/server.example.com@EXAMPLE.COM
    10/31/2019 15:51:08  11/01/2019 15:51:02  krbtgt/EXAMPLE.COM@EXAMPLE.COM
  2. Test the Directory Server and SSSD configuration on every replica by retrieving user information.

    [root@server ~]# ipa user-show admin
      User login: admin
      Last name: Administrator
      Home directory: /home/admin
      Login shell: /bin/bash
      Principal alias: admin@EXAMPLE.COM
      UID: 1965200000
      GID: 1965200000
      Account disabled: False
      Password: True
      Member of groups: admins, trust admins
      Kerberos keys available: True
  3. Test the CA server on every CA replica with the ipa cert-show command.

    [root@server ~]# ipa cert-show 1
      Issuing CA: ipa
      Certificate: MIIEgjCCAuqgAwIBAgIjoSIP...
      Subject: CN=Certificate Authority,O=EXAMPLE.COM
      Issuer: CN=Certificate Authority,O=EXAMPLE.COM
      Not Before: Thu Oct 31 19:43:29 2019 UTC
      Not After: Mon Oct 31 19:43:29 2039 UTC
      Serial number: 1
      Serial number (hex): 0x1
      Revoked: False

4.3. Recovering from a VM snapshot to establish a new IdM environment

If the Certificate Authority (CA) replica from a restored Virtual Machine (VM) snapshot is unable to replicate with other servers, create a new IdM environment from the VM snapshot.

To establish a new IdM environment, isolate the VM server, create additional replicas from it, and switch IdM clients to the new environment.

Prerequisites

Procedure

  1. Boot the desired snapshot of the CA replica VM.
  2. Isolate the restored server from the rest of the current deployment by removing all of its replication topology segments.

    1. First, display all domain replication topology segments.

      [root@restored-CA-replica ~]# ipa topologysegment-find
      Suffix name: domain
      ------------------
      8 segments matched
      ------------------
        Segment name: new_segment
        Left node: restored-CA-replica.example.com
        Right node: server2.example.com
        Connectivity: both
      
      ...
      
      ----------------------------
      Number of entries returned 8
      ----------------------------
    2. Next, delete every domain topology segment involving the restored server.

      [root@restored-CA-replica ~]# ipa topologysegment-del
      Suffix name: domain
      Segment name: new_segment
      -----------------------------
      Deleted segment "new_segment"
      -----------------------------
    3. Finally, perform the same actions with any ca topology segments.

      [root@restored-CA-replica ~]# ipa topologysegment-find
      Suffix name: ca
      ------------------
      1 segments matched
      ------------------
        Segment name: ca_segment
        Left node: restored-CA-replica.example.com
        Right node: server4.example.com
        Connectivity: both
      ----------------------------
      Number of entries returned 1
      ----------------------------
      
      [root@restored-CA-replica ~]# ipa topologysegment-del
      Suffix name: ca
      Segment name: ca_segment
      -----------------------------
      Deleted segment "ca_segment"
      -----------------------------
  3. Install a sufficient number of IdM replicas from the restored server to handle the deployment load. There are now two disconnected IdM deployments running in parallel.
  4. Switch the IdM clients to use the new deployment by hard-coding references to the new IdM replicas. See Adjusting IdM clients during recovery.
  5. Stop and uninstall IdM servers from the previous deployment. See Uninstalling an IdM server.

Verification steps

  1. Test the Kerberos server on every new replica by successfully retrieving a Kerberos ticket-granting ticket as an IdM user.

    [root@server ~]# kinit admin
    Password for admin@EXAMPLE.COM:
    
    [root@server ~]# klist
    Ticket cache: KCM:0
    Default principal: admin@EXAMPLE.COM
    
    Valid starting       Expires              Service principal
    10/31/2019 15:51:37  11/01/2019 15:51:02  HTTP/server.example.com@EXAMPLE.COM
    10/31/2019 15:51:08  11/01/2019 15:51:02  krbtgt/EXAMPLE.COM@EXAMPLE.COM
  2. Test the Directory Server and SSSD configuration on every new replica by retrieving user information.

    [root@server ~]# ipa user-show admin
      User login: admin
      Last name: Administrator
      Home directory: /home/admin
      Login shell: /bin/bash
      Principal alias: admin@EXAMPLE.COM
      UID: 1965200000
      GID: 1965200000
      Account disabled: False
      Password: True
      Member of groups: admins, trust admins
      Kerberos keys available: True
  3. Test the CA server on every new CA replica with the ipa cert-show command.

    [root@server ~]# ipa cert-show 1
      Issuing CA: ipa
      Certificate: MIIEgjCCAuqgAwIBAgIjoSIP...
      Subject: CN=Certificate Authority,O=EXAMPLE.COM
      Issuer: CN=Certificate Authority,O=EXAMPLE.COM
      Not Before: Thu Oct 31 19:43:29 2019 UTC
      Not After: Mon Oct 31 19:43:29 2039 UTC
      Serial number: 1
      Serial number (hex): 0x1
      Revoked: False