Chapter 2. Restoring a containerized undercloud
The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:
- You have re-installed the latest version of Red Hat Enterprise Linux 8.
- The hardware layout is the same.
- The hostname and undercloud settings of the machine are the same.
-
The backup archive has been copied to the
root
directory.
Procedure
-
Log into your undercloud as the
root
user. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:
[root@director ~]# subscription-manager register
Attach the Red Hat OpenStack Platform entitlement:
[root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456
Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:
[root@director ~]# subscription-manager repos --disable=* [root@director ~]# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-16-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
Perform an update on your system to make sure you have the latest base system packages:
[root@director ~]# dnf update -y [root@director ~]# reboot
Ensure the time on your undercloud is synchronized. For example:
[root@director ~]# dnf install -y chrony [root@director ~]# systemctl start chronyd [root@director ~]# systemctl enable chronyd
-
Copy the undercloud backup archive to the undercloud’s
root
directory. The following steps useundercloud-backup-$TIMESTAMP.tar
as the filename, where $TIMESTAMP is a Bash variable for the timestamp on the archive. Install the database server and client tools:
[root@director ~]# dnf install -y mariadb mariadb-server
Start the database:
[root@director ~]# systemctl start mariadb
Increase the allowed packets to accommodate the size of our database backup:
[root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
Extract the database and database configuration from the archive:
[root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/config-data/mysql/etc/my.cnf.d/galera.cnf [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sql
Restore the database backup:
[root@director ~]# mysql -u root < /root/undercloud-all-databases.sql
Extract a temporary version of the root configuration file:
[root@director ~]# tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnf
Get the old root database password:
[root@director ~]# OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")
Reset the root database password:
[root@director ~]# mysqladmin -u root password "$OLDPASSWORD"
Copy the root configuration file from the temporary location:
[root@director ~]# mv root/.my.cnf . [root@director ~]# rmdir root
Get a list of old user permissions:
[root@director ~]# mysql -e 'select host, user, password from mysql.user;'
Remove the old user permissions for each host listed. For example:
[root@director ~]# HOST="192.0.2.1" [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs) [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done [root@director ~]# mysql -e 'flush privileges'
Perform this for all users accessing through the host IP and any host ("
%
").
The IP address in the HOST parameter is the undercloud’s IP address in the control plane.
Stop the database:
[root@director ~]# systemctl stop mariadb
Create the
stack
user:[root@director ~]# useradd stack
Set a password for the user:
[root@director ~]# passwd stack
Disable password requirements when using
sudo
:[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack [root@director ~]# chmod 0440 /etc/sudoers.d/stack
Restore the
stack
user home directory:# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stack
Install the
python3-policycoreutils
package:[root@director ~]# dnf -y install python3-policycoreutils
Restore the
glance
data:[root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance
Restore the
swift
data:[root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node
If using SSL in the undercloud, refresh the CA certificates:
[root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/instack-certs/undercloud.pem [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/ca-trust/source/anchors/* [root@director ~]# restorecon -R /etc/pki [root@director ~]# semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?" [root@director ~]# restorecon -R /etc/pki/instack-certs [root@director ~]# update-ca-trust extract
Switch to the
stack
user:[root@director ~]# su - stack [stack@director ~]$
Install the
python3-tripleoclient
package:$ sudo dnf install -y python3-tripleoclient ceph-ansible
Run the undercloud installation command. Ensure that you run it in the
stack
user’s home directory:[stack@director ~]$ openstack undercloud install
When the install completes, the undercloud automatically restores its connection to the overcloud. The nodes continue to poll OpenStack Orchestration (heat) for pending tasks.