Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Appendix A. Restoring the undercloud

The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:

  • You have re-installed the latest version of Red Hat Enterprise Linux 7.
  • The hardware layout is the same.
  • The hostname and undercloud settings of the machine are the same.
  • The backup archive has been copied to the root directory.


  1. Log into your undercloud as the root user.
  2. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:

    [root@director ~]# subscription-manager register
  3. Attach the Red Hat OpenStack Platform entitlement:

    [root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456
  4. Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:

    [root@director ~]# subscription-manager repos --disable=*
    [root@director ~]# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms
  5. Perform an update on your system to ensure that you have the latest base system packages:

    [root@director ~]# yum update -y
    [root@director ~]# reboot
  6. Ensure that the time on your undercloud is synchronized. For example:

    [root@director ~]# yum install -y ntp
    [root@director ~]# systemctl start ntpd
    [root@director ~]# systemctl enable ntpd
    [root@director ~]# ntpdate
    [root@director ~]# systemctl restart ntpd
  7. Copy the undercloud backup archive to the undercloud’s root directory. The following steps use undercloud-backup-$TIMESTAMP.tar as the filename, where $TIMESTAMP is a Bash variable for the timestamp on the archive.
  8. Install the database server and client tools:

    [root@director ~]# yum install -y mariadb mariadb-server
  9. Start the database:

    [root@director ~]# systemctl start mariadb
    [root@director ~]# systemctl enable mariadb
  10. Increase the allowed packets to accommodate the size of our database backup:

    [root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
  11. Extract the database and database configuration from the archive:

    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/my.cnf.d/*server*.cnf
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sql
  12. Restore the database backup:

    [root@director ~]# mysql -u root < /root/undercloud-all-databases.sql
  13. Extract a temporary version of the root configuration file:

    [root@director ~]# tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnf
  14. Get the old root database password:

    [root@director ~]# OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")
  15. Reset the root database password:

    [root@director ~]# mysqladmin -u root password "$OLDPASSWORD"
  16. Move the root configuration file from the temporary directory to the root directory:

    [root@director ~]# mv ~/root/.my.cnf ~/.
    [root@director ~]# rmdir ~/root
  17. Get a list of old user permissions:

    [root@director ~]# mysql -e 'select host, user, password from mysql.user;'
  18. Remove the old user permissions for each host listed. For example:

    [root@director ~]# HOST=""
    [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs)
    [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done
    [root@director ~]# for USER in $USERS ; do mysql -e "drop user $USER" || true ;done
    [root@director ~]# mysql -e 'flush privileges'

    Perform this for all users accessing through the host IP and any host (“%”).


    The IP address in the HOST parameter is the undercloud’s IP address in control plane.

  19. Restart the database:

    [root@director ~]# systemctl restart mariadb
  20. Create the stack user:

    [root@director ~]# useradd stack
  21. Set a password for the user:

    [root@director ~]# passwd stack
  22. Disable password requirements when using sudo:

    [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
    [root@director ~]# chmod 0440 /etc/sudoers.d/stack
  23. Restore the stack user home directory:

    # tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stack
  24. Install the policycoreutils-python package:

    [root@director ~]# yum -y install policycoreutils-python
  25. Install the openstack-glance package and restore its data and file permissions:

    [root@director ~]# yum install -y openstack-glance
    [root@director ~]# tar --xattrs --xattrs-include='*.*' -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance/images
    [root@director ~]# chown -R glance: /var/lib/glance/images
    [root@director ~]# restorecon -R /var/lib/glance/images
  26. Install the openstack-swift package and restore its data and file permissions:

    [root@director ~]# yum install -y openstack-swift
    [root@director ~]# tar --xattrs --xattrs-include='*.*' -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node
    [root@director ~]# chown -R swift: /srv/node
    [root@director ~]# restorecon -R /srv/node
  27. Install the openstack-keystone package and restore its configuration data:

    [root@director ~]# yum -y install openstack-keystone
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/keystone
    [root@director ~]# restorecon -R /etc/keystone
  28. Install the openstack-heat and restore configuration:

    [root@director ~]# yum install -y openstack-heat*
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/heat
    [root@director ~]# restorecon -R /etc/heat
  29. Install puppet and restore its configuration data:

    [root@director ~]# yum install -y puppet hiera
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/puppet/hieradata/
  30. If you use SSL in the undercloud, refresh the CA certificates. Depending on your undercloud configuration, use either the steps for user-provided certificates or the steps for the auto-generated certificates:

    • If the undercloud is configured with user-provided certificates, complete the following steps:

      1. Extract the certificates:

        [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/instack-certs/undercloud.pem
        [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/ca-trust/source/anchors/*
      2. Restore the SELinux contexts and manage the file system labelling:

        [root@director ~]# restorecon -R /etc/pki
        [root@director ~]# semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?"
        [root@director ~]# restorecon -R /etc/pki/instack-certs
      3. Update the certificates:

        [root@director ~]# update-ca-trust extract
    • If you use certmonger to auto-generate certificates for the undercloud, complete the following steps:

      1. Extract certificates, CA certificate and certmonger files:

        [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/certmonger/*
        [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/tls/*
        [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/ca-trust/source/anchors/*
      2. Restore the SELinux contexts:

        [root@director ~]# restorecon -R /etc/pki
        [root@director ~]# restorecon -R /var/lib/certmonger
      3. Remove the /var/lib/certmonger/lock file:

        [root@director ~]# rm -f /var/lib/certmonger/lock
  31. Switch to the stack user:

    [root@director ~]# su - stack
    [stack@director ~]$
  32. Install the python-tripleoclient package:

    $ sudo yum install -y python-tripleoclient
  33. Run the undercloud installation command. Ensure that you run it in the stack user’s home directory:

    [stack@director ~]$ openstack undercloud install

    When the install completes, the undercloud automatically restores its connection to the overcloud. The nodes continue to poll OpenStack Orchestration (heat) for pending tasks.