Chapter 4. Executing the backup procedure

Before you perform a fast forward upgrade, back up the undercloud and the overcloud control plane nodes so that you can restore them to their previous state if an error occurs.

Note

Before you back up the undercloud and overcloud, ensure that you do not perform any operations on the overcloud from the undercloud.

4.1. Performing prerequisite tasks before backing up the undercloud

Do not perform an undercloud backup when you deploy the undercloud or when you make changes to an existing undercloud. To prevent data corruptions, confirm that there are no stack failures, ongoing tasks, and that all OpenStack services except for mariadb are stopped before you back up the undercloud node.

Procedure

  1. List failures for all available stacks:

    (undercloud) [stack@undercloud-0 ~]$ source stackrc && for i in `openstack stack list -c 'Stack Name' -f value`;  do openstack stack failures list $i; done
  2. Verify that there are no ongoing tasks in the cloud:

    (undercloud) [stack@undercloud-0 ~]$ openstack stack list --nested | grep -v "_COMPLETE"

    If the command returns no results, there are no ongoing tasks.

  3. Stop all OpenStack services in the cloud:

    # systemctl stop tripleo_*
  4. Start the tripleo_mysql service:

    # systemctl start tripleo_mysql
  5. Verify that the tripleo_mysql service is running:

    # systemctl status tripleo_mysql

4.2. Backing up the undercloud

To back up the undercloud node, you must log in as the root user on the undercloud node. As a precaution, you must back up the database to ensure that you can restore it.

Prerequisites

Procedure

  1. Locate the database password.

    [root@undercloud stack]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)
  2. Back up the databases:

    # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql
    # podman exec mysql bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql
  3. Stop the mariadb database service:

    [root@undercloud stack]# systemctl stop tripleo_mysql
  4. Create the backup:

    [root@undercloud stack]# rear -d -v mkbackup

    You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

4.3. Backing up the control plane

To back up the control plane, you must first stop the pacemaker cluster and all containers operating on the control plane nodes. Do not operate the stack to ensure state consistency. After you complete the backup procedure, start the pacemaker cluster and the containers.

As a precaution, you must back up the database to ensure that you can restore the database after you restart the pacemaker cluster and containers.

Back up the control plane nodes simultaneously.

Prerequisites

Procedure

  1. Locate the database password:

    [heat-admin@overcloud-controller-x ~]# PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)
  2. Back up the databases:

    [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -p$PASSWORD -s -N -e | sed 's/$/;/' " > openstack-backup-mysql-grants.sql
    [heat-admin@overcloud-controller-x ~]# podman exec galera-bundle-podman-X bash -c "mysql -uroot -p$PASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -p$PASSWORD --single-transaction --databases" > openstack-backup-mysql.sql
  3. On one of control plane nodes, stop the pacemaker cluster:

    Important

    Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure.

    [heat-admin@overcloud-controller-x ~]# pcs cluster stop --all
  4. On each control plane node, stop the containers.

    1. Stop the containers:

      [heat-admin@overcloud-controller-x ~]# systemctl stop tripleo_*
    2. Stop the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mon@$(hostname -s)
    3. Stop the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# sudo systemctl stop ceph-mgr@$(hostname -s)
  5. To back up the control plane, run the following command as root in the command line interface of each control plane node:

    [heat-admin@overcloud-controller-x ~]# rear -d -v mkbackup

    You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory.

    Note

    When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following:

    WARNING: tar ended with return code 1 and below output:
    ---snip---
    tar: /var/spool/postfix/public/qmgr: socket ignored
    ...
    ...
    This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.
  6. When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster and the containers:

    On one of the control plane nodes, enter the following command:

    [heat-admin@overcloud-controller-x ~]# pcs cluster start --all
  7. On each control plane node, start the containers.

    1. Start the ceph-mon@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mon@$(hostname -s)
    2. Start the ceph-mgr@controller.service container:

      [heat-admin@overcloud-controller-x ~]# systemctl start ceph-mgr@$(hostname -s)