Appendix A. Restoring the undercloud

The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:

  • You have re-installed the latest version of Red Hat Enterprise Linux 7.
  • The hardware layout is the same.
  • The hostname and undercloud settings of the machine are the same.
  • The backup archive has been copied to the root directory.

Procedure

  1. Log into your undercloud as the root user.
  2. Create the stack user:

    [root@director ~]# useradd stack
  3. Set a password for the user:

    [root@director ~]# passwd stack
  4. Disable password requirements when using sudo:

    [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
    [root@director ~]# chmod 0440 /etc/sudoers.d/stack
  5. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:

    [root@director ~]# subscription-manager register
  6. Attach the Red Hat OpenStack Platform entitlement:

    [root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456
  7. Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

    [root@director ~]# subscription-manager repos --disable=*
    [root@director ~]# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-13-rpms
  8. Perform an update on your system to make sure you have the latest base system packages:

    [root@director ~]# yum update -y
    [root@director ~]# reboot
  9. Ensure the time on your undercloud is synchronized. For example:

    [root@director ~]# yum install -y ntp
    [root@director ~]# systemctl start ntpd
    [root@director ~]# systemctl enable ntpd
    [root@director ~]# ntpdate pool.ntp.org
    [root@director ~]# systemctl restart ntpd
  10. Create a temporary directory for the backup

    [root@director ~]# mkdir /var/tmp/undercloud_backup
  11. Extract the filesystem backup archive into the temporary directory:

    [root@director ~]# tar -xvf /root/undercloud-backup-[timestamp].tar -C /var/tmp/undercloud_backup --xattrs || true
  12. Install rsync:

    [root@director ~]# yum -y install rsync
  13. Synchronize the following directories with backup content:

    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/home/stack/ /home/stack
    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/etc/haproxy/ /etc/haproxy/
    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/etc/pki/instack-certs/ /etc/pki/instack-certs/
    [root@director ~]# mkdir -p /etc/puppet/hieradata/
    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/etc/puppet/hieradata/ /etc/puppet/hieradata/
    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/srv/node/ /srv/node/
    [root@director ~]# rsync -a -X /var/tmp/undercloud_backup/var/lib/glance/ /var/lib/glance/
  14. Install the openstack-keystone package and synchronize its configuration data:

    [root@director ~]# yum -y install openstack-keystone
    [root@director ~]# rsync -a /var/tmp/undercloud_backup/etc/keystone/ /etc/keystone/
  15. Install the policycoreutils-python package:

    [root@director ~]# yum -y install policycoreutils-python
  16. If using SSL in the undercloud, refresh the CA certificates:

    [root@director ~]# semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?"
    [root@director ~]# restorecon -R /etc/pki/instack-certs
    [root@director ~]# update-ca-trust extract
  17. Install the database server and client tools:

    [root@director ~]# yum install -y mariadb mariadb-server python-tripleoclient
  18. Start the database:

    [root@director ~]# systemctl start mariadb
    [root@director ~]# systemctl enable mariadb
  19. Increase the allowed packets to accommodate the size of our database backup:

    [root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
  20. Restore the database backup:

    [root@director ~]# mysql -u root < /var/tmp/undercloud_backup/root/undercloud-all-databases.sql
  21. Restart Mariadb to refresh the permissions from the backup file:

    [root@director ~]# systemctl restart mariadb
  1. Get a list of old user permissions:

    [root@director ~]# mysql -e 'select host, user, password from mysql.user;'
  2. Remove the old user permissions for each host listed. For example:

    [root@director ~]# HOST="192.0.2.1"
    [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs)
    [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done
    [root@director ~]# mysql -e 'flush privileges'
  3. Install the openstack-glance package and restore its file permissions:

    [root@director ~]# yum install -y openstack-glance
    [root@director ~]# chown -R glance: /var/lib/glance/images
  4. Install the openstack-swift packages and restore its file permissions:

    [root@director ~]# yum install -y openstack-swift
    [root@director ~]# chown -R swift: /srv/node
  5. Switch to the new stack user:

    [root@director ~]# su - stack
    [stack@director ~]$
  6. Run the undercloud installation command. Ensure to run it in the stack user’s home directory:

    [stack@director ~]$ openstack undercloud install
  7. Wait until the install completes. The undercloud automatically restores its connection to the overcloud. The nodes will continue to poll OpenStack Orchestration (heat) for pending tasks.