Chapter 4. Executing the backup procedure

Procedure

1. List failures for all available stacks:

   (undercloud) [stack@undercloud-0 ~]$ source stackrc && for i in `openstack stack list -c 'Stack Name' -f value`;  do openstack stack failures list $i; done

2. Verify that there are no ongoing tasks in the cloud:

   (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -v "_COMPLETE"

   If the command returns no results, there are no ongoing tasks.

3. Stop all OpenStack services in the cloud:

   # systemctl stop tripleo_*

4. Start the tripleo_mysql service:

   # systemctl start tripleo_mysql

5. Verify that the tripleo_mysql service is running:

   # systemctl status tripleo_mysql

Procedure

1. Locate the database password.

   [root@undercloud stack]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

2. Back up the databases:

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

3. Stop the mariadb database service:

   [root@undercloud stack]# systemctl stop tripleo_mysql

4. Create the backup:

   [root@undercloud stack]# rear -d -v mkbackup

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

Procedure

1. Locate the database password:

   [heat-admin@overcloud-controller-x ~]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)

2. Back up the databases:

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql

   [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql

3. On one of control plane nodes, stop the pacemaker cluster:

   Important

   Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

   [heat-admin@overcloud-controller-x ~]# pcs cluster stop --all

4. On each control plane node, stop the containers.

   1. Stop the containers:

      [heat-admin@overcloud-controller-x ~]# systemctl stop tripleo_*

   2. Stop the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mon@$(hostname -s)

   3. Stop the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mgr@$(hostname -s)

5. To back up the control plane, run the following command as root in the command line interface of each control plane node:

   [heat-admin@overcloud-controller-x ~]# rear -d -v mkbackup

   You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

   Note

   When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following:

   WARNING: tar ended with return code 1 and below output:
   ---snip---
   tar: /var/spool/postfix/public/qmgr: socket ignored
   ...
   ...
   This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.

6. When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster and the containers:

   On one of the control plane nodes, enter the following command:

   [heat-admin@overcloud-controller-x ~]# pcs cluster start --all

7. On each control plane node, start the containers.

   1. Start the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mon@$(hostname -s)

   2. Start the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mgr@$(hostname -s)