Chapter 2. Restore

This section describes how to restore the undercloud used in the Red Hat Enterprise Linux OpenStack Platform Director.

2.1. Restore the Undercloud

The following restore process assumes you are recovering a failed undercloud node, and you need to reinstall it from scratch. It assumes that the hardware layout is identical, and the hostname and undercloud settings of the machine will also be identical.

Once the machine is installed and is in a clean state, re-enable all the subscriptions/repositories needed to install and run director. Run the following commands as the root user:

1. Install the mariadb server:

# yum install -y mariadb-server

2. Restore the MariaDB configuration file and database backup, then start the MariaDB server and load the backup data:

a. As the root user, restore the MariaDB files:

# tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/my.cnf.d/server.cnf
# tar -xzC / -f undercloud-backup-$DATE.tar.gz root/undercloud-all-databases.sql

b. Edit /etc/my.cnf.d/server.cnf and comment out the bind-address entry.

c. Start the mariadb service:

# systemctl start mariadb
# cat /root/undercloud-all-databases.sql | mysql

d. Clean up certain permissions (to be recreated later):

# for i in ceilometer glance heat ironic keystone neutron nova tuskar;do mysql -e "drop user $i";done
# mysql -e 'flush privileges'

3. Create the stack user account:

# sudo useradd stack
# sudo passwd stack  # specify a password
# echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
# sudo chmod 0440 /etc/sudoers.d/stack

4. Restore the stack user home directory:

# tar -xzC / -f undercloud-backup-$DATE.tar.gz home/stack

5. Install the swift and glance base packages, and then restore their data:

# yum install -y openstack-glance openstack-swift
# tar -xzC / -f undercloud-backup-$DATE.tar.gz srv/node var/lib/glance/images

6. Confirm the data is owned by the correct user:

# chown -R swift: /srv/node
# chown -R glance: /var/lib/glance/images

7. Restore your HAproxy SSL certificates:

# tar -xzC / -f undercloud-backup-$DATE.tar.gz etc/keystone/ssl
# semanage fcontext -a -t etc_t "/etc/keystone/ssl(/.*)?"
# restorecon -R /etc/keystone/ssl

8. Re-run the undercloud installation as the stack user, making sure to run it in the stack user home directory:

# su - stack
$ sudo yum install -y python-rdomanager-oscplugin

9. Confirm that the hostname is correctly set in /etc/hosts.

10. Reinstall the undercloud:

$ openstack undercloud install

2.2. Reconnect the restored undercloud to the overcloud

Having completed the steps above, the undercloud can be expected to automatically restore its connection to the overcloud. The nodes will continue to poll Orchestration (heat) for pending tasks, using a simple HTTP request issued every few seconds.

2.3. Validate the Completed Restore

Use the following commands to perform a healthcheck of your newly restored environment:

2.3.1. Check Identity Service (Keystone) Operation

This step validates Identity Service operations by querying for a list of users.

# source overcloudrc
# keystone user-list

When run from the controller, the output of this command should include a list of users created in your environment. This action demonstrates that keystone is running and successfully authenticating user requests. For example:

# keystone user-list
+----------------------------------+------------+---------+----------------------+
|                id                |    name    | enabled |        email         |
+----------------------------------+------------+---------+----------------------+
| 9e47bb53bb40453094e32eccce996828 |   admin    |   True  |    root@localhost    |
| 9fe2466f88cc4fa0ba69e59b47898829 | ceilometer |   True  | ceilometer@localhost |
| 7a40d944e55d422fa4e85daf47e47c42 |   cinder   |   True  |   cinder@localhost   |
| 3d2ed97538064f258f67c98d1912132e |    demo    |   True  |                      |
| 756e73a5115d4e9a947d8aadc6f5ac22 |   glance   |   True  |   glance@localhost   |
| f0d1fcee8f9b4da39556b78b72fdafb1 |  neutron   |   True  |  neutron@localhost   |
| e9025f3faeee4d6bb7a057523576ea19 |    nova    |   True  |    nova@localhost    |
| 65c60b1278a0498980b2dc46c7dcf4b7 |   swift    |   True  |   swift@localhost    |
+----------------------------------+------------+---------+----------------------+

2.3.2. Check the OpenStack Services

Run the openstack-status command to view the status of OpenStack services:

# openstack-status