Undercloud and Control Plane Back Up and Restore

Red Hat OpenStack Platform 16.1

Procedures for backing up and restoring the undercloud and the overcloud control plane during updates and upgrades

Abstract

This guide explains how to install and configure Relax-and-Recover (ReaR) on the undercloud and overcloud control plane nodes; how to back up the undercloud and Control Plane nodes before updates and upgrades; and, how to restore the undercloud and Control Plane nodes if an error occurs while performing updates or upgrades.

Chapter 1. Introduction to undercloud and control plane back up and restore

Undercloud and Control Plane Back Up and Restore describes the tasks that are required to back up the state of the Red Hat OpenStack Platform 16.1 undercloud and overcloud controller nodes, also known as control plane nodes, before updates and upgrades. You can use the created backups to restore the undercloud and overcloud control plane nodes to their previous state if an error occurs during an update or upgrade.

1.1. About the ReaR disaster recovery solution

The tasks described in the Undercloud and Control Plane Back Up and Restore guide use the open source Relax and Recover (ReaR) disaster recovery solution that is written in Bash. You can use ReaR to create bootable images of the latest state of the undercloud or control plane nodes, or to back up specific files.

ReaR supports the following boot media formats:

  • ISO
  • USB
  • eSATA
  • PXE

The examples in this document were tested using the ISO bootable files format.

ReaR can use the following protocols to transport files:

  • HTTP/HTTPS
  • SSH/SCP
  • FTP/SFTP
  • NFS
  • CIFS (SMB)

For the purposes of backing up and restoring the Red Hat OpenStack Platform 16.1 undercloud and overcloud control plane nodes, the examples in this document were tested using NFS.

1.2. ReaR backup management options

You can use ReaR with internal and external backup management options.

Internal backup management

You can use ReaR with the following internal backup options:

  • tar
  • rsync

External backup management

External backup management options include open source and proprietary solutions. You can use ReaR with the following open source solutions:

  • Bacula
  • Bareos

You can use ReaR with the following proprietary solutions:

  • EMC NetWorker (Legato)
  • HP DataProtector
  • IBM Tivoli Storage Manager (TSM)
  • Symantec NetBackup

Chapter 2. Configuring the backup node

Before you can create a backup of the undercloud or control plane nodes, you must configure the backup node. You can install and configure an NFS server on the backup node using the backup-and-restore Ansible role.

Procedure

  1. On the undercloud node, source the undercloud credentials: 

    [stack@undercloud-0 ~]$ source stackrc
(undercloud) [stack@undercloud ~]$

  2. On the undercloud node, create an inventory file for the backup node and replace the <IP_ADDRESS> and <USER> with the values that apply to your environment: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF'> ~/nfs-inventory.yaml
[BACKUP_NODE]
serverX ansible_host=<IP_ADDRESS>
ansible_user=<USER>
EOF

  3. On the undercloud node, create the following Ansible playbook and replace <BACKUP_NODE> with the host name of the backup node: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_nfs_setup.yaml
# Playbook
# Substitute <BACKUP_NODE> with the host name of your backup node.
- become: true
  hosts: <BACKUP_NODE>
  name: Setup NFS server for ReaR
  roles:
  - role: backup-and-restore
EOF

  4. On the undercloud node, enter the following ansible-playbook command, to configure the backup node: 

    (undercloud) [stack@undercloud ~]$ ansible-playbook \
    -v -i ~/nfs-inventory.yaml \
    --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
    --become \
    --become-user root \
    --tags bar_setup_nfs_server \
    ~/bar_nfs_setup.yaml

Chapter 3. Installing ReaR on the undercloud and control plane nodes

Before creating a backup of the undercloud and control plane nodes, you must install the Relax and Recover (ReaR) packages on the undercloud node and on each of the controller nodes.

To install ReaR using the backup-and-restore Ansible role, complete the following procedures:

  1. Section 3.1, “Installing ReaR on the undercloud node”
  2. Section 3.2, “Installing ReaR on the control plane nodes”

3.1. Installing ReaR on the undercloud node

To create a backup of the undercloud node, you must install and configure Relax and Recover (ReaR) on the undercloud.

Prerequisites

Procedure

  1. On the undercloud node, source the undercloud credentials and use the tripleo-ansible-inventory command to generate a static inventory file that contains hosts and variables for all the overcloud nodes: 

    [stack@undercloud-0 ~]$ source stackrc
(undercloud) [stack@undercloud ~]$ tripleo-ansible-inventory \
--ansible_ssh_user heat-admin \
--static-yaml-inventory /home/stack/tripleo-inventory.yaml

  2. On the undercloud node, create the following Ansible playbook: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_setup-undercloud.yaml
# Playbook
# Installing and configuring ReaR on the undercloud node
- become: true
  hosts: undercloud
  name: Install ReaR
  roles:
  - role: backup-and-restore
EOF

  3. On the undercloud node, enter the following ansible-playbook command to install Rear: 

    (undercloud) [stack@undercloud ~]$ ansible-playbook \
    -v -i ~/tripleo-inventory.yaml \
    --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
    --become \
    --become-user root \
    --tags bar_setup_rear \
    ~/bar_rear_setup-undercloud.yaml

3.2. Installing ReaR on the control plane nodes

To create a backup of the overcloud control plane, you must install and configure Relax and Recover (ReaR) on each of the control plane nodes.

Prerequisites

Procedure

  1. On the undercloud node, create the following Ansible playbook: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_setup-controller.yaml
# Playbook
# Install and configuring ReaR on the control plane nodes
- become: true
  hosts: Controller
  name: Install ReaR
  roles:
  - role: backup-and-restore
EOF

  2. On the undercloud node, enter the following ansible-playbook command to install Rear on the control plane nodes: 

    (undercloud) [stack@undercloud ~]$ ansible-playbook \
    -v -i ~/tripleo-inventory.yaml \
    --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
    --become \
    --become-user root \
    --tags bar_setup_rear \
    ~/bar_rear_setup-controller.yaml

Chapter 4. Creating a backup of the undercloud and control plane nodes

To create a backup of the undercloud and control plane nodes using the backup-and-restore Ansible role, complete the following procedures:

  1. Section 4.1, “Creating a backup of the undercloud node”
  2. Section 4.2, “Creating a backup of the control plane nodes”

4.1. Creating a backup of the undercloud node

You can use the backup-and-restore Ansible role to create a backup of the undercloud node.

Prerequisites

Procedure

  1. On the undercloud node, create the following Ansible playbook: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_create_restore_images-undercloud.yaml
# Playbook
# Using ReaR on the undercloud node.
- become: true
  hosts: undercloud
  name: Create the recovery images for the undercloud
  roles:
  - role: backup-and-restore
EOF

  2. To create a backup of the undercloud node, enter the following ansible-playbook command: 

    (undercloud) [stack@undercloud ~]$ ansible-playbook \
    -v -i ~/tripleo-inventory.yaml \
    --extra="ansible_ssh_common_args=-o StrictHostKeyChecking=no" \
    --become \
    --become-user root \
    --tags bar_create_recover_image \
    ~/bar_rear_create_restore_images-undercloud.yaml

4.2. Creating a backup of the control plane nodes

You can use the backup-and-restore Ansible role to create a backup of the control plane nodes.

Prerequisites

Procedure

  1. On the undercloud node, create the following Ansible playbook: 

    (undercloud) [stack@undercloud ~]$ cat <<'EOF' > ~/bar_rear_create_restore_images-controller.yaml
# Playbook
# Using ReaR on the control plane nodes.
- become: true
  hosts: Controller
  name: Create the recovery images for the control plane
  roles:
  - role: backup-and-restore
EOF

  2. On the undercloud node, enter the following ansible-playbook command, to create a backup of the control plane nodes: 

    (undercloud) [stack@undercloud ~]$ ansible-playbook \
    -v -i ~/tripleo-inventory.yaml \
    --extra="ansible_ssh_common_args='-o StrictHostKeyChecking=no'" \
    --become \
    --become-user root \
    --tags bar_create_recover_image \
    ~/bar_rear_create_restore_images-controller.yaml

Chapter 5. Restoring the undercloud and control plane nodes

If an error occurs during an update or upgrade, you can restore either the undercloud or overcloud control plane nodes, or both to their previous state using backups.

To restore the undercloud and control plane nodes using backups, complete the following procedures:

  1. Section 5.1, “Restoring the undercloud node”
  2. Section 5.2, “Restoring the control plane nodes”

5.1. Restoring the undercloud node

If an error occurs during an update or upgrade, you can restore the undercloud node to its previous state using the backup ISO image that you created using ReaR. You can find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access.

Prerequisites

Procedure

  1. Power off the undercloud node. Ensure that the undercloud node is powered off completely before you proceed.
  2. Boot the undercloud node with the backup ISO image.
  3. When the Relax-and-Recover boot menu displays, select Recover <UNDERCLOUD_NODE> where <UNDERCLOUD_NODE> is the name of your undercloud node.

  4. Log in as the root user and restore the node:

    The following message displays: 

    Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
RESCUE <UNDERCLOUD_NODE>:~ # rear recover

    When the undercloud node restoration process completes, the console displays the following message: 

    Finished recovering your system
Exiting rear recover
Running exit tasks

  5. When the command line interface is available, power off the node: 

    RESCUE <UNDERCLOUD_NODE>:~ #  poweroff

    On boot up, the node resumes its previous state.

5.2. Restoring the control plane nodes

If an error occurs during an update or upgrade, you can restore the control plane nodes to their previous state using the backup ISO image that you have created using ReaR.

To restore the control plane, you must restore all control plane nodes to ensure state consistency.

You can find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access.

Prerequisites

Procedure

  1. Power off each control plane node. Ensure that the control plane nodes are powered off completely before you proceed.
  2. Boot each control plane node with the corresponding backup ISO image.
  3. When the Relax-and-Recover boot menu displays, on each control plane node, select Recover <CONTROL PLANE NODE>. Replace <CONTROL PLANE NODE> with the name of the corresponding control plane node.

  4. On each control plane node, log in as the root user and restore the node:

    The following message displays: 

    Welcome to Relax-and-Recover. Run "rear recover" to restore your system!
RESCUE <CONTROL PLANE NODE>:~ # rear recover

    When the control plane node restoration process completes, the console displays the following message: 

    Finished recovering your system
Exiting rear recover
Running exit tasks

  5. When the command line interface is available on each control plane node, power off the node: 

    RESCUE <CONTROL PLANE NODE>:~ #  poweroff
  6. Set the boot sequence to the normal boot device. On boot up, the node resumes its previous state.

  7. To ensure that the services are running correctly, check the status of pacemaker. Log in to a Controller node as the root user and enter the following command: 

    # pcs status
  8. To view the status of the overcloud, use Tempest. For more information about Tempest, see Chapter 4 of the OpenStack Integration Test Suite Guide.

Legal Notice

Copyright © 2020 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.