Chapter 2. Restore the Undercloud
This section describes how to restore the undercloud used in the Red Hat OpenStack Platform Director.
2.1. Restoring the undercloud
The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:
- You have re-installed the latest version of Red Hat Enterprise Linux 7.
- The hardware layout is the same.
- The hostname and undercloud settings of the machine are the same.
-
The backup archive has been copied to the
rootdirectory.
Procedure
-
Log into your undercloud as the
rootuser. Create the
stackuser:[root@director ~]# useradd stack
Set a password for the user:
[root@director ~]# passwd stack
Disable password requirements when using
sudo:[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack [root@director ~]# chmod 0440 /etc/sudoers.d/stack
Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:
[root@director ~]# sudo subscription-manager register
Attach the Red Hat OpenStack Platform entitlement:
[root@director ~]# sudo subscription-manager attach --pool=Valid-Pool-Number-123456
Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:
[root@director ~]# sudo subscription-manager repos --disable=* [root@director ~]# sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-13-rpms
Perform an update on your system to make sure you have the latest base system packages:
[root@director ~]# sudo yum update -y [root@director ~]# sudo reboot
Ensure the time on your undercloud is synchronized. For example:
[root@director ~]# sudo yum install -y ntp [root@director ~]# sudo systemctl start ntpd [root@director ~]# sudo systemctl enable ntpd [root@director ~]# sudo ntpdate pool.ntp.org [root@director ~]# sudo systemctl restart ntpd
Create a temporary directory for the backup
[root@director ~]# mkdir /var/tmp/undercloud_backup
Extract the filesystem backup archive into the temporary directory:
[root@director ~]# sudo tar -xvf /root/undercloud-backup-[timestamp].tar -C /var/tmp/undercloud_backup --xattrs || true
Install
rsync:[root@director ~]# sudo yum -y install rsync
Synchronize the following directories with backup content:
[root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/home/stack/ /home/stack [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/etc/haproxy/ /etc/haproxy/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/etc/pki/instack-certs/ /etc/pki/instack-certs/ [root@director ~]# sudo mkdir -p /etc/puppet/hieradata/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/etc/puppet/hieradata/ /etc/puppet/hieradata/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/srv/node/ /srv/node/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/var/lib/glance/ /var/lib/glance/
Install the
openstack-keystonepackage and synchronize its configuration data:[root@director ~]# sudo yum -y install openstack-keystone [root@director ~]# sudo rsync -a /var/tmp/undercloud_backup/etc/keystone/ /etc/keystone/
Install the
policycoreutils-pythonpackage:[root@director ~]# sudo yum -y install policycoreutils-python
If using SSL in the undercloud, refresh the CA certificates:
[root@director ~]# sudo semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?" [root@director ~]# sudo restorecon -R /etc/pki/instack-certs [root@director ~]# sudo update-ca-trust extract
Install the database server and client tools:
[root@director ~]# sudo yum install -y mariadb mariadb-server python-tripleoclient
Start the database:
[root@director ~]# sudo systemctl start mariadb [root@director ~]# sudo systemctl enable mariadb
Increase the allowed packets to accommodate the size of our database backup:
[root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
Restore the database backup:
[root@director ~]# mysql -u root < /var/tmp/undercloud_backup/root/undercloud-all-databases.sql
Restart Mariadb to refresh the permissions from the backup file:
[root@director ~]# sudo systemctl restart mariadb
Get a list of old user permissions:
[root@director ~]# mysql -e 'select host, user, password from mysql.user;'
Remove the old user permissions for each host listed. For example:
[root@director ~]# HOST="192.0.2.1" [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs) [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done [root@director ~]# mysql -e 'flush privileges'
Install the
openstack-glancepackage and restore its file permissions:[root@director ~]# sudo yum install -y openstack-glance [root@director ~]# sudo chown -R glance: /var/lib/glance/images
Install the
openstack-swiftpackages and restore its file permissions:[root@director ~]# sudo yum install -y openstack-swift [root@director ~]# sudo chown -R swift: /srv/node
Switch to the new
stackuser:[root@director ~]# su - stack [stack@director ~]$
Run the undercloud installation command. Ensure to run it in the
stackuser’s home directory:[stack@director ~]$ openstack undercloud install
- Wait until the install completes. The undercloud automatically restores its connection to the overcloud. The nodes will continue to poll OpenStack Orchestration (heat) for pending tasks.
Synchronize the container data with backup content:
[root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/var/lib/docker/ /var/lib/docker/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/var/lib/registry/ /var/lib/registry/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/etc/docker/ /etc/docker/ [root@director ~]# sudo rsync -a -X /var/tmp/undercloud_backup/etc/docker-distribution/ /etc/docker-distribution/ [root@director ~]# sudo cp /var/tmp/undercloud_backup/etc/sysconfig/docker* /etc/sysconfig/. [root@director ~]# sudo systemctl restart docker docker-distribution
2.2. Validate the Completed Restore
Use the following commands to perform a healthcheck of your newly restored environment:
2.2.1. Check Identity Service (Keystone) Operation
This step validates Identity Service operations by querying for a list of users.
# source overcloudrc # openstack user list
When run from the controller, the output of this command should include a list of users created in your environment. This action demonstrates that keystone is running and successfully authenticating user requests. For example:
# openstack user list +----------------------------------+------------+---------+----------------------+ | id | name | enabled | email | +----------------------------------+------------+---------+----------------------+ | 9e47bb53bb40453094e32eccce996828 | admin | True | root@localhost | | 9fe2466f88cc4fa0ba69e59b47898829 | ceilometer | True | ceilometer@localhost | | 7a40d944e55d422fa4e85daf47e47c42 | cinder | True | cinder@localhost | | 3d2ed97538064f258f67c98d1912132e | demo | True | | | 756e73a5115d4e9a947d8aadc6f5ac22 | glance | True | glance@localhost | | f0d1fcee8f9b4da39556b78b72fdafb1 | neutron | True | neutron@localhost | | e9025f3faeee4d6bb7a057523576ea19 | nova | True | nova@localhost | | 65c60b1278a0498980b2dc46c7dcf4b7 | swift | True | swift@localhost | +----------------------------------+------------+---------+----------------------+
