Procedure

1. On the undercloud node, source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc
   (undercloud) [stack@undercloud ~]$

2. On the undercloud node, create an inventory file for the backup node and replace the <IP_ADDRESS> and <USER> with the values that apply to your environment:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml
   [BACKUP_NODE]
   serverX ansible_host=<IP_ADDRESS> ansible_user=<USER>
   EOF

3. On the undercloud node, create the following Ansible playbook and replace <BACKUP_NODE> with the host name of the backup node:

   (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_nfs_setup.yaml
   # Playbook
   # Substitute <BACKUP_NODE> with the host name of your backup node.
   - become: true
     hosts: <BACKUP_NODE>
     name: Setup NFS server for ReaR
     roles:
     - role: backup-and-restore
   EOF

4. On the undercloud node, enter the following ansible-playbook command, to configure the backup node:

   (undercloud) [stack@undercloud ~]$ ansible-playbook \
       -v -i ~/nfs-inventory.yaml \
       --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
       --become \
       --become-user root \
       --tags bar_setup_nfs_server \
       ~/bar_nfs_setup.yaml

1. Section 3.1, “Installing ReaR on the undercloud node”
2. Section 3.2, “Installing ReaR on the control plane nodes”

1. Section 4.1, “Creating a backup of the undercloud node”
2. Section 4.2, “Creating a backup of the control plane nodes”

1. Section 5.1, “Restoring the undercloud node”
2. Section 5.2, “Restoring the control plane nodes”

Procedure

1. On the undercloud node, enter the following command to create the backup script:

   [stack@undercloud ~]$ cat <<'EOF' > /home/stack/execute-rear-cron.sh
   
   #!/bin/bash
   
   OWNER="stack"
   TODAY=`date +%Y%m%d`
   FILE="/var/log/rear-cron.${TODAY}"
   sudo touch ${FILE}
   sudo chown ${OWNER}:${OWNER} ${FILE}
   
   CURRENTTIME=`date`
   echo "[$CURRENTTIME] rear start" >> ${FILE}
   /usr/bin/ansible-playbook -v -i /home/stack/tripleo-inventory.yaml --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" --become --become-user root --tags bar_create_recover_image --extra="tripleo_backup_and_restore_service_manager=false" /home/stack/bar_rear_create_restore_images.yaml 2>&1 >> ${FILE}
   CURRENTTIME=`date`
   echo "[$CURRENTTIME] rear end" >> ${FILE}
   EOF

2. Set executable privileges for the /home/stack/execute-rear-cron.sh script:

   [stack@undercloud ~]$ chmod 755 /home/stack/execute-rear-cron.sh

3. Edit the crontab file with the crontab -e command and use an editor of your choice to add the following cron job. Ensure you save the changes to the file:

   [stack@undercloud ~]# $ crontab -e
   #adding the following line
   0 0 * * * /home/stack/execute-rear-cron.sh

   The /home/stack/execute-rear-cron.sh script is scheduled to be executed by the stack user at midnight.

4. To verify that the cron job is scheduled, enter the following command:

   [stack@undercloud ~]$ crontab -l

   The command output displays the scheduled cron jobs:

   0 0 * * * /home/stack/execute-rear-cron.sh

Procedure

1. On the backup node, export the NFS directory to host the Ceph backups. Replace <IP_ADDRESS/24> with the IP address and subnet mask of the network:

   [root@backup ~]# cat >> /etc/exports << EOF
   /ceph_backups <IP_ADDRESS/24>(rw,sync,no_root_squash,no_subtree_check)
   EOF

2. On the undercloud node, source the undercloud credentials and run the following script:

   # source stackrc

   #! /bin/bash
   for i in `openstack server list -c Name -c Networks -f value | grep controller | awk -F'=' '{print $2}' | awk -F' ' '{print $1}'`; do ssh -q heat-admin@$i 'sudo systemctl stop ceph-mon@$(hostname -s) ceph-mgr@$(hostname -s)'; done

   To verify that the ceph-mgr@controller.service container has stopped, enter the following command:

   [heat-admin@overcloud-controller-x ~]# sudo podman ps | grep ceph

3. On the undercloud node, source the undercloud credentials and run the following script. Replace <BACKUP_NODE_IP_ADDRESS> with the IP address of the backup node:

   # source stackrc

   #! /bin/bash
   for i in `openstack server list -c Name -c Networks -f value | grep controller | awk -F'=' '{print $2}' | awk -F' ' '{print $1}'`; do ssh -q heat-admin@$i 'sudo mkdir /ceph_backups'; done
   
   #! /bin/bash
   for i in `openstack server list -c Name -c Networks -f value | grep controller | awk -F'=' '{print $2}' | awk -F' ' '{print $1}'`; do ssh -q heat-admin@$i 'sudo mount -t nfs  <BACKUP_NODE_IP_ADDRESS>:/ceph_backups /ceph_backups'; done
   
   #! /bin/bash
   for i in `openstack server list -c Name -c Networks -f value | grep controller | awk -F'=' '{print $2}' | awk -F' ' '{print $1}'`; do ssh -q heat-admin@$i 'sudo mkdir /ceph_backups/$(hostname -s)'; done
   
   #! /bin/bash
   for i in `openstack server list -c Name -c Networks -f value | grep controller | awk -F'=' '{print $2}' | awk -F' ' '{print $1}'`; do ssh -q heat-admin@$i 'sudo tar -zcv --xattrs-include=*.* --xattrs  --xattrs-include=security.capability --xattrs-include=security.selinux --acls -f /ceph_backups/$(hostname -s)/$(hostname -s).tar.gz  /var/lib/ceph'; done

4. On the node that you want to restore, complete the following tasks:

   1. Power off the node before you proceed.
   2. Restore the node with the ReaR backup file that you have created during the backup process. The file is located in the /ceph_backups directory of the backup node.
   3. From the Relax-and-Recover boot menu, select Recover <CONTROL_PLANE_NODE>, where <CONTROL_PLANE_NODE> is the name of the control plane node.

   4. At the prompt, enter the following command:

      RESCUE <CONTROL_PLANE_NODE> :~ # rear recover

      When the image restoration process completes, the console displays the following message:

   Finished recovering your system
   Exiting rear recover
   Running exit tasks

5. For the node that you want to restore, copy the Ceph backup from the /ceph_backups directory into the /var/lib/ceph directory:

   1. Identify the system mount points:

      RESCUE <CONTROL_PLANE_NODE>:~# df -h
      Filesystem      Size  Used Avail Use% Mounted on
      devtmpfs         16G     0   16G   0% /dev
      tmpfs            16G     0   16G   0% /dev/shm
      tmpfs            16G  8.4M   16G   1% /run
      tmpfs            16G     0   16G   0% /sys/fs/cgroup
      /dev/vda2        30G   13G   18G  41% /mnt/local

      The /dev/vda2 file system is mounted on /mnt/local.

   2. Create a temporary directory:

      RESCUE <CONTROL_PLANE_NODE>:~ # mkdir /tmp/restore
      RESCUE <CONTROL_PLANE_NODE>:~ # mount -v -t nfs -o rw,noatime <BACKUP_NODE_IP_ADDRESS>:/ceph_backups /tmp/restore/

   3. On the control plane node, remove the existing /var/lib/ceph directory:

      RESCUE <CONTROL_PLANE_NODE>:~ # rm -rf /mnt/local/var/lib/ceph/*

   4. Restore the previous Ceph maps. Replace <CONTROL_PLANE_NODE> with the name of your control plane node:

      RESCUE <CONTROL_PLANE_NODE>:~ # tar -xvC /mnt/local/ -f /tmp/restore/<CONTROL_PLANE_NODE>/<CONTROL_PLANE_NODE>.tar.gz --xattrs --xattrs-include='*.*' var/lib/ceph

   5. Verify that the files are restored:

      RESCUE <CONTROL_PLANE_NODE>:~ # ls -l
      total 0
      drwxr-xr-x 2 root 107 26 Jun 18 18:52 bootstrap-mds
      drwxr-xr-x 2 root 107 26 Jun 18 18:52 bootstrap-osd
      drwxr-xr-x 2 root 107 26 Jun 18 18:52 bootstrap-rbd
      drwxr-xr-x 2 root 107 26 Jun 18 18:52 bootstrap-rgw
      drwxr-xr-x 3 root 107 31 Jun 18 18:52 mds
      drwxr-xr-x 3 root 107 31 Jun 18 18:52 mgr
      drwxr-xr-x 3 root 107 31 Jun 18 18:52 mon
      drwxr-xr-x 2 root 107  6 Jun 18 18:52 osd
      drwxr-xr-x 3 root 107 35 Jun 18 18:52 radosgw
      drwxr-xr-x 2 root 107  6 Jun 18 18:52 tmp

6. Power off the node:

   RESCUE <CONTROL_PLANE_NODE> :~ #  poweroff

7. Power on the node. The node resumes its previous state.