Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 3. Director-Based Environments: Performing Upgrades to Major Versions

Warning

Before performing an upgrade to the latest major version, ensure the undercloud and overcloud are updated to the latest minor versions. This includes both OpenStack Platform services and the base operating system. For the process on performing a minor version update, see "Updating the Environment" in the Red Hat OpenStack Platform 7 Director Installation and Usage guide. Performing a major version upgrade without first performing a minor version update can cause failures in the upgrade process.

Warning

With High Availaibility for Compute instances (or Instance HA, as described in High Availability for Compute Instances), upgrades or scale-up operations are not possible. Any attempts to do so will fail.

If you have Instance HA enabled, disable it before performing an upgrade or scale-up. To do so, perform a rollback as described in Rollback.

This chapter explores how to upgrade your environment. This includes upgrading aspects of both the Undercloud and Overcloud. This upgrade process provides a means for you to move to the next major version. In this case, it is a upgrade from Red Hat OpenStack Platform 7 to Red Hat OpenStack Platform 8.

This procedure for both situations involves the following workflow:

  1. Update the Red Hat OpenStack Platform director packages
  2. Update the Overcloud images on the Red Hat OpenStack Platform director
  3. Update the Overcloud stack and its packages using the Red Hat OpenStack Platform director
Important

Make sure to read the information in Section 3.1, “Important Pre-Upgrade Notes” before attempting a version upgrade.

3.1. Important Pre-Upgrade Notes

Make sure you have read the following notes before upgrading your environment.

  • Upgrade in Red Hat OpenStack Platform director requires full testing with specific configurations before performed on any live production environment. Red Hat has tested most use cases and combinations that are offered as standard options through the director but, due to the number of possible combinations, this can never be a fully exhaustive list. In addition, if the configuration has been modified from the standard deployment, either manually or through post configuration hooks, testing the upgrade feature in a non-production environment becomes even more critical. Therefore, we advise you to:

    • Perform a backup of your Undercloud node before starting any steps in the upgrade procedure. See the Back Up and Restore Red Hat OpenStack Platform guide for backup procedures.
    • Run the upgrade procedure in a test environment that includes all of the changes made before running the procedure in your production environment.
    • Please contact Red Hat and request guidance and assistance on the upgrade process before proceeding if you feel uncomfortable about performing this upgrade.
  • The upgrade process outlined in this section only accommodates customizations through the director. If you customized an Overcloud feature outside of director then disable the feature, upgrade the Overcloud, and re-enable the feature after the upgrade completes. This means the customized feature is unavailable until the completion of the entire upgrade.

  • Red Hat OpenStack Platform director 8 can manage pervious Overcloud versions. See the support matrix below for information.

    Table 3.1. Support Matrix for Red Hat OpenStack Platform director 8

    Version

    Overcloud Updating

    Overcloud Deploying

    Overcloud Scaling

    Red Hat OpenStack Platform 7

    7.0.4 and newer

    7.0.4 and newer

    7.0.4 and newer

    Red Hat OpenStack Platform 8

    All versions

    All versions

    All versions

  • If using managing and older Overcloud version, use the following Heat template collections:

    • For Red Hat OpenStack Platform 7: /usr/share/openstack-tripleo-heat-templates/kilo/

      For example: 

      $ openstack overcloud deploy -templates /usr/share/openstack-tripleo-heat-templates/kilo/ [OTHER_OPTIONS]

  • If managing a Red Hat OpenStack Platform 7 Overcloud, set the RabbitMQ password to the version 7 default in the /home/stack/tripleo-overcloud-passwords file: 

    OVERCLOUD_RABBITMQ_PASSWORD=guest

  • If using an environment file for Satellite registration, make sure to update the following parameters in the environment file:

    • rhel_reg_repos - Repositories to enable for your Overcloud, including the new Red Hat OpenStack Platform 8 repositories. See Section 1.2, “Repository Requirements” for repositories to enable.
    • rhel_reg_activation_key - The new activation key for your Red Hat OpenStack Platform 8 repositories.
    • rhel_reg_sat_repo - A new parameter that defines the repository containing Red Hat Satellite 6’s management tools, such as katello-agent. Make sure to add this parameter if registering to Red Hat Satellite 6.
  • The default timezone for Red Hat OpenStack Platform 8 is now UTC. The default timezone for Red Hat OpenStack Platform 7 was EST. If necessary, include an environment file to specify the timezone.
  • If using an external load balancer, update your load balancing settings to accommodate new services in Red Hat OpenStack Platform 8. For a full list of services and example configuration, see "Services Configuration Reference" in the External Load Balancing for the Overcloud guide.
  • Make sure that you have upgraded your undercloud and overcloud to the latest minor release of Red Hat OpenStack Platform 7 and Red Hat Enterprise Linux 7 before attempting a major upgrade to Red Hat OpenStack Platform 8. See "Updating the Environment" in the Red Hat OpenStack Platform 7 Director Installation and Usage guide for instructions on performing a package update to your undercloud and overcloud. If the kernel updates to the latest version, perform a reboot so that new kernel parameters take effect.
  • Apply a version lock to libvirt as described in Solutions.

3.2. Upgrading the Director

Important

Make sure to read the information in Section 3.1, “Important Pre-Upgrade Notes” before attempting any step in the following procedure.

Log into the director as the stack user and stop the main OpenStack Platform services: 

$ sudo systemctl stop 'openstack-*' 'neutron-*' httpd
Note

This causes a short period of downtime for the undercloud. The overcloud is still functional during the undercloud update.

To update the director packages to the latest major version, change the OpenStack Platform repository from the old version to the new version. For example: 

$ sudo subscription-manager repos --disable=rhel-7-server-openstack-7.0-rpms --disable=rhel-7-server-openstack-7.0-director-rpms
$ sudo subscription-manager repos --enable=rhel-7-server-openstack-8-rpms --enable=rhel-7-server-openstack-8-director-rpms

This sets yum to use the latest repositories. Use yum to update the director: 

$ sudo yum upgrade

Some OpenStack services might fail after yum update completes. This is expected behavior. The Undercloud’s upgrade command corrects the configuration of these services.

The director uses the openstack undercloud upgrade command to upgrade the Undercloud environment. Run the upgrade command: 

$ openstack undercloud upgrade

This refreshes the director’s configuration and populates any settings that are unset since the version change. Running this command does not delete any stored data, such Overcloud stack data or data for existing nodes in your environment.

Once the update completes, check the director’s OpenStack services: 

$ sudo systemctl list-units openstack-*
Note

The openstack-keystone might appear as a failed service. This is because the service now runs through the httpd service as a WSGI application. The openstack-keystone service is safe to disable after updating director packages and running openstack undercloud upgrade.

To finalize the update, verify the existence of your Overcloud and its nodes: 

$ source ~/stackrc
$ openstack server list
$ ironic node-list
$ heat stack-list

Be aware of the following notes after upgrading the Overcloud to Red Hat OpenStack Platform 8:

  • Underclouds using SSL might experience a loss of access to VIPs during the upgrade. If so, restart the keepalived service on the Undercloud: 

    $ systemctl restart keepalived

  • The Undercloud’s admin user might require an additional role (_member_) not included with Red Hat OpenStack Platform 8. This role is important for Overcloud communication. Check for this role: 

    $ keystone role-list

    If the role does not exist, create it: 

    $ keystone role-create --name _member_

    Add the role to the admin user on the admin tenant: 

    $ keystone user-role-add --user admin --role _member_ --tenant admin

  • If using customized core Heat templates, make sure to check for differences between the updated core Heat templates and your current set. Red Hat provides updates to the Heat template collection over subsequent releases. Using a modified template collection can lead to a divergence between your custom copy and the original copy in /usr/share/openstack-tripleo-heat-templates. Run the following command to see differences between your custom Heat template collection and the updated original version: 

    # diff -Nary /usr/share/openstack-tripleo-heat-templates/ ~/templates/my-overcloud/

    Make sure to either apply these updates to your custom Heat template collection, or create a new copy of the templates in /usr/share/openstack-tripleo-heat-templates/ and apply your customizations.

3.3. Upgrading the Overcloud Images on the Director

Important

Make sure to read the information in Section 3.1, “Important Pre-Upgrade Notes” before attempting any step in the following procedure.

This procedure ensures you have the latest images for node discovery and Overcloud deployment. The new images from the rhosp-director-images and rhosp-director-images-ipa packages are already updated from the Undercloud upgrade.

Remove any existing images from the images directory on the stack user’s home (/home/stack/images), then copy the new image archives: 

$ rm -rf ~/images/*
$ cp /usr/share/rhosp-director-images/overcloud-full-latest-8.0.tar ~/images/.
$ cp /usr/share/rhosp-director-images/ironic-python-agent-latest-8.0.tar ~/images/.

Extract the archives: 

$ cd ~/images
$ for tarfile in *.tar; do tar -xf $tarfile; done

Import the latest images into the director and configure nodes to use the new images 

$ openstack overcloud image upload --update-existing --image-path ~/images/.
$ openstack baremetal configure boot

To finalize the image update, verify the existence of the new images: 

$ openstack image list
$ ls -l /httpboot

The director is now upgraded with the latest images.

Important

Make sure the Overcloud image version corresponds to the Undercloud version.

3.4. Upgrading the Overcloud

Important

Make sure to read the information in Section 3.1, “Important Pre-Upgrade Notes” before attempting any step in the following procedure.

This section details the steps required to upgrade the Overcloud. Make sure to follow each section in order and only apply the sections relevant to your environment.

This process requires you to run your original openstack overcloud deploy command multiple times to provide a staged method of upgrading. Each time you run the command, you include a different upgrade environment file along with your existing environment files. These new upgrade environments files are:

  • major-upgrade-pacemaker-init.yaml - Provides the initialization for the upgrade. This includes updating the Red Hat OpenStack Platform repositories on each node in your Overcloud and provides special upgrade scripts to certain nodes.
  • major-upgrade-pacemaker.yaml - Provides an upgrade for the Controller nodes.
  • major-upgrade-pacemaker-converge.yaml - The finalization for the Overcloud upgrade. This aligns the resulting upgrade to match the contents for the director’s latest Heat template collection.

In between these deployment commands, you run the upgrade-non-controller.sh script on various node types. This script updates the packages on a non-Controller node.

Workflow

The Overcloud upgrade process uses the following workflow:

  1. Run your deployment command including the major-upgrade-pacemaker-init.yaml environment file.
  2. Run the upgrade-non-controller.sh on each Object Storage node.
  3. Run your deployment command including the major-upgrade-pacemaker.yaml environment file.
  4. Run the upgrade-non-controller.sh on each Compute node.
  5. Run the upgrade-non-controller.sh on each Ceph Storage node.
  6. Run your deployment command including the major-upgrade-pacemaker-converge.yaml environment file.

3.4.1. Including the Management Network

If using custom NIC templates from Red Hat OpenStack Platform 7, add the ManagementSubnetIp parameter to the parameters section of your NIC templates. For example: 

parameters:
  ManagementIpSubnet: # Only populated when including environments/network-management.yaml
    default: ''
    description: IP address/subnet on the management network
    type: string

3.4.2. Installing the Upgrade Scripts

This step installs scripts on each non-Controller node. These script perform the major version package upgrades and configuration. Each script differs depending on the node type. For example, Compute nodes receive different upgrade scripts to Ceph Storage nodes.

This initialization step also updates enabled repositories on all overcloud nodes. This means you do not need to disable old repositories and enable new repositories manually.

Run the openstack overcloud deploy from your Undercloud and include the major-upgrade-pacemaker-init.yaml environment file. Make sure you also include all custom environment files relevant to your environment, such as network isolation and storage.

This following is an example of the openstack overcloud deploy command with the added file. For example: 

$ openstack overcloud deploy --templates \
  --control-scale 3 \
  --compute-scale 3 \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml  \
  -e /home/stack/templates/network_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml

Wait until the Overcloud updates with the new environment file’s configuration.

3.4.3. Upgrading Object Storage Nodes

The director uses the upgrade-non-controller.sh command to run the upgrade script passed to each non-Controller node from the major-upgrade-pacemaker-init.yaml environment file. For this step, you upgrade each Object Storage node with the following command: 

$ for NODE in `nova list | grep swift | awk -F "|" '{ print $2 }'` ; do upgrade-non-controller.sh --upgrade $NODE ; done

Wait until each Object Storage node completes its upgrade.

3.4.4. Upgrading Controller Nodes

Upgrading the Controller nodes involves including another environment file (major-upgrade-pacemaker.yaml) that provides a full upgrade to Controller nodes running high availability tools.

Run the openstack overcloud deploy from your Undercloud and include the major-upgrade-pacemaker.yaml environment file. Make sure you also include all custom environment files relevant to your environment, such as network isolation and storage.

This following is an example of the openstack overcloud deploy command with the added file. For example: 

$ openstack overcloud deploy --templates \
  --control-scale 3 \
  --compute-scale 3 \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
  -e network_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml

Wait until the Overcloud updates with the new environment file’s configuration.

Important

This step disables the Neutron server and L3 Agent during the Controller upgrade. This means floating IP address are unavailable during this step.

Important

If the Overcloud stack fails during this step, log into one of your Controller nodes, run sudo pcs cluster start, then rerun openstack overcloud deploy on the director.

3.4.5. Upgrading Compute Nodes

The director uses the upgrade-non-controller.sh command to run the upgrade script passed to each non-Controller node from the major-upgrade-pacemaker-init.yaml environment file.

Obtain a list of Compute nodes and their UUIDs: 

$ nova list | grep "compute"

We perform the following task on each node individually to ensure zero downtime. Select a Compute node to upgrade.

  1. Migrate its workload. See "Migrating VMs from an Overcloud Compute Node" in the Red Hat OpenStack Platform Director Installation and Usage Guide.

  2. Ensure the Compute service for the chosen node is disabled: 

    $ source ~/overcloudrc
$ nova service-list
$ nova service-disable [hostname] nova-compute

  3. Upgrade each Compute node with the following command: 

    $ source ~/stackrc
$ upgrade-non-controller.sh --upgrade NODE_UUID

    Replace NODE_UUID with the UUID of the chosen Compute node

  4. Wait until the Compute node completes its upgrade. After the upgrade completes, make sure the Compute nodes are enabled using the following command: 

    $ source ~/overcloudrc
$ nova service-list
$ nova service-enable [hostname] nova-compute

Repeat these steps on each Compute node.

3.4.6. Upgrading Ceph Storage Nodes

The director uses the upgrade-non-controller.sh command to run the upgrade script passed to each non-Controller node from the major-upgrade-pacemaker-init.yaml environment file. For this step, you upgrade each Ceph Storage node with the following command:

Upgrade each Ceph Storage nodes: 

$ for NODE in `nova list | grep ceph | awk -F "|" '{ print $2 }'` ; do upgrade-non-controller.sh --upgrade $NODE ; done

3.4.7. Finalizing the Upgrade

The director needs to run through the upgrade finalization to ensure the Overcloud stack is synchronized with the current Heat template collection. This involves an environment file (major-upgrade-pacemaker-converge.yaml), which you include using the openstack overcloud deploy command.

Run the openstack overcloud deploy from your Undercloud and include the major-upgrade-pacemaker-converge.yaml environment file. Make sure you also include all custom environment files relevant to your environment, such as network isolation and storage.

This following is an example of the openstack overcloud deploy command with the added file. For example: 

$ openstack overcloud deploy --templates \
  --control-scale 3 \
  --compute-scale 3 \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
  -e network_env.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml

Wait until the Overcloud updates with the new environment file’s configuration.

This completes the Overcloud upgrade procedure.

3.4.8. Post-Upgrade Notes

Be aware of the following notes after upgrading the Overcloud to Red Hat OpenStack Platform 8:

  • The Compute nodes might report a failure with neutron-openvswitch-agent. If this occurs, log into each Compute node and restart the service. For example: 

    $ sudo systemctl restart neutron-openvswitch-agent
  • The update process does not reboot any nodes in the Overcloud automatically. If required, perform a reboot manually after the update command completes. Make sure to reboot cluster-based nodes (such as Ceph Storage nodes and Controller nodes) individually and wait for the node to rejoin the cluster. For Ceph Storage nodes, check with the ceph health and make sure the cluster status is HEALTH OK. For Controller nodes, check with the pcs resource and make sure all resources are running for each node.

  • In some circumstances, the corosync service might fail to start on IPv6 environments after rebooting Controller nodes. This is due to Corosync starting before the Controller node configures the static IPv6 addresses. In these situations, restart Corosync manually on the Controller nodes: 

    $ sudo systemctl restart corosync

  • If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes: 

    $ sudo pcs property set stonith-enabled=true

  • The next time you update or scale the Overcloud stack (i.e. running the openstack overcloud deploy command), you need to reset the identifier that triggers package updates in the Overcloud. Add a blank UpdateIdentifier parameter to an environment file and include it when you run the openstack overcloud deploy command. The following is an example of such an environment file: 

    parameter_defaults:
  UpdateIdentifier: