Chapter 2. Planning and preparation for an in-place upgrade

Before you conduct an in-place upgrade of your OpenStack Platform environment, create a plan for the upgrade and accommodate any potential obstacles that might block a successful upgrade.

2.1. Familiarize yourself with Red Hat OpenStack Platform 16.1

Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 16.1 to help you understand the resulting environment and any potential version-to-version changes what might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 16.1, follow these suggestions:

2.2. High level changes in Red Hat OpenStack Platform 16.1

The following high-level changes occur during the upgrade to Red Hat OpenStack Platform 16.1:

  • OpenStack Platform director 16.1 configures the overcloud using an Ansible-driven method called config-download. This replaces the standard heat-based configuration method. Director still uses heat to orchestrate provisioning operations.
  • The director installation uses the same method as the overcloud deployment. Therefore, the undercloud also uses openstack-tripleo-heat-templates as a blueprint for installing and configuring each service.
  • The undercloud runs OpenStack services in containers.
  • The undercloud pulls and stores container images through a new method. Instead of pulling container images before deploying the overcloud, the undercloud pulls all relevant container images during the deployment process.
  • The overcloud deployment process includes an Advanced Subscription Management method to register nodes. This method incorporates an Ansible role to register OpenStack Platform nodes. The new method also applies different subscriptions to different node roles if necessary.
  • The overcloud now uses Open Virtual Network (OVN) as the default ML2 mechanism driver. It is possible to migrate your Open vSwitch (OVS) service to OVN, which you perform after the completion of a successful upgrade.
  • The undercloud and overcloud both run on Red Hat Enterprise Linux 8.
  • openstack-tripleo-heat-templates includes a unified composable service template collection in the deployment directory. This directory now includes templates with merged content from both the containerized service and Puppet-based composable service templates.
  • The OpenStack Data Processing service (sahara) is no longer supported.

    Important

    If you have sahara enabled in your Red Hat OpenStack Platform 13 environment, do not continue with this upgrade and contact Red Hat Global Support Services.

  • The OpenStack Telemetry components are deprecated in favor of the Service Telemetry Framework (STF).

2.3. Changes in Red Hat Enterprise Linux 8

The undercloud and overcloud both run on Red Hat Enterprise Linux 8. This includes new tools and functions relevant to the undercloud and overcloud:

  • The undercloud and overcloud use the Red Hat Container Toolkit. Instead of docker to build and control the container lifecycle, Red Hat Enterprise Linux 8 includes buildah to build new container images and podman for container management.
  • Red Hat Enterprise Linux 8 does not include the docker-distribution package. The undercloud now includes a private HTTP registry to provide container images to overcloud nodes.
  • The upgrade process from Red Hat Enterprise Linux 7 to 8 uses the leapp tool.
  • Red Hat Enterprise Linux 8 does not use the ntp service. Instead, Red Hat Enterprise Linux 8 uses chronyd.
  • Red Hat Enterprise Linux 8 includes new versions of high availability tools.

The Red Hat OpenStack Platform 16.1 uses Red Hat Enterprise Linux 8.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to Red Hat Enterprise Linux 8.2.

For more information about the key differences between Red Hat Enterprise Linux 7 and 8, see "Considerations in adopting RHEL 8". For general information about Red Hat Enterprise linux 8, see Product Documentation for Red Hat Enterprise Linux 8.

2.4. Leapp upgrade usage in Red Hat OpenStack Platform

The long-life Red Hat OpenStack Platform upgrade also requires an upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8. Red Hat Enterprise Linux 7 includes a tool named leapp, which performs the upgrade to Red Hat Enterprise Linux 8. Both the undercloud and overcloud use a separate process for performing the operating system upgrade.

Undercloud process

Run the leapp upgrade manually before you run the openstack undercloud upgrade command. The undercloud upgrade includes instructions for performing the leapp upgrade.

Overcloud process

The overcloud upgrade framework automatically runs the leapp upgrade.

Limitations

For information of potential limitations that might affect your upgrade, see the following sections from the Upgrading from RHEL 7 to RHEL 8 guide:

In particular, you cannot perform a Leapp upgrade on nodes that use encryption of the whole disk or a partition, such as LUKS encryption, or file-system encryption. This limitation affects Ceph OSD nodes that you have configured with the dmcrypt: true parameter.

If any known limitations affect your environment, seek advice from the Red Hat Technical Support Team.

2.5. Supported upgrade scenarios

Before proceeding with the upgrade, check that your overcloud is supported.

Note

If you are uncertain whether a particular scenario not mentioned in these lists is supported, seek advice from the Red Hat Technical Support Team.

Supported scenarios

The following in-place upgrade scenarios are tested and supported.

  • Standard environments with default role types: Controller, Compute, and Ceph Storage OSD
  • Split-Controller composable roles
  • Ceph Storage composable roles
  • Hyper-Converged Infrastructure: Compute and Ceph Storage OSD services on the same node
  • Environments with Network Functions Virtualization (NFV) technologies: Single-root input/output virtualization (SR-IOV) and Data Plane Development Kit (DPDK)

Technology preview scenarios

The framework for upgrades is considered a Technology Preview when you use it in conjunction with these features, and therefore is not fully supported by Red Hat. You should only test this scenario in a proof-of-concept environment and not upgrade in a production environment. For more information about Technology Preview features, see Scope of Coverage Details.

  • Edge and Distributed Compute Node (DCN) scenarios

Untested and unsupported

The following in-place upgrade scenarios are either untested or unsupported. If your overcloud configuration matches any of the scenarios on this list, postpone your upgrade and seek advice from the Red Hat Technical Support Team.

  • Environments with Instance HA enabled

2.6. Considerations for upgrading with external Ceph deployments

If you have deployed a Red Hat Ceph Storage system separately and then used director to deploy and configure OpenStack, you can use the Red Hat OpenStack Platform framework for upgrades to perform an in-place upgrade with external Ceph deployments. This scenario is different from upgrading a Ceph cluster that was deployed using director.

The differences that you must take into account when planning and preparing for an in-place upgrade with external Ceph deployments are the following:

  1. Before you can upgrade your Red Hat OpenStack Platform deployment from version 13 to version 16.1, you must upgrade your Red Hat Ceph Storage cluster from version 3 to version 4. For more information, see Upgrading a Red Hat Ceph Storage cluster in the Red Hat Ceph Storage 4 Installation Guide.
  2. After you upgrade your Red Hat Ceph Storage cluster from version 3 to version 4, Red Hat OpenStack Platform 13 might still run RHCSv3 client components, however these are compatible against the RHCSv4 cluster.
  3. You can follow the upgrade path described in the Framework For Upgrades (13 to 16.1) document, and where applicable, you must complete the conditional steps that support this particular scenario. A conditional step starts with the following statement: "If you are upgrading with external Ceph deployments".
  4. When you upgrade with external Ceph deployments, you install RHCSv4 ceph-ansible as part of the overcloud upgrade process. When you upgrade a Ceph cluster that was deployed using director, you install RHCSv4 ceph-ansible after the overcloud upgrade process is complete.

2.7. Known issues that might block an upgrade

Review the following known issues that might affect a successful upgrade.

BZ#1852360 - swift_rsync and swift_rsync_healthcheck moved in failed state after undercloud upgrade
The containerized tripleo_swift_rsync service on the Red Hat OpenStack Platform 16.1 undercloud reports a failed state. This is due to a conflict with the Red Hat OpenStack Platform 16.1 version running the OpenStack Object Storage (swift) rsync daemon through xinetd. This guide includes xinetd in the list of packages that you must remove before to the undercloud upgrade. Red Hat aims to resolve this issue in the next 16.1 minor release update.
BZ#1866479 - container-tools module stream not enabled correctly
The Red Hat Enterprise Linux 8 AppStream repository uses container-tools:rhel8 as a default dnf module. The overcloud and undercloud require a change to the container-tools:2.0 module to use the correct version of Podman for Red Hat OpenStack Platform 16.1 support. You change this module manually on the undercloud but director must change this module on the overcloud automatically during the upgrade process. This guide includes a workaround that uses the UpgradeInitCommand parameter to perform this module change on the overcloud.
BZ#1871834 - ovn controllers cannot connect ovn dbs during 13-16.1 FFU upgrade

Due to an VIP address change for Open Virtual Network (OVN), OVN controllers can no longer connect to the OVN database during an upgrade. To work around this issue:

  1. After upgrading Controller nodes, obtain the new OVN VIP:

    [root@controller-0 ~]# ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g'

    This produces output in the following format.

    tcp:<NewOvnVIP>:6642
  2. Before you start the Compute upgrade process, log in to each Compute node and set the OVN VIP:

    [root@compute-0 ~]# sudo docker exec -uroot ovn_controller ovs-vsctl set Open_Vswitch . external_ids:ovn-remote="tcp:<NewOvnVIP>:6642"

Red Hat aims to resolve this issue in the next 16.1 minor release update.

BZ#1885212 - ovn-controllers can’t connect to SB DB in hybrid state

+

OpenStack Platform 13 environments that use OVN will experience OVN southbound database connection issues during the upgrade. ovn-controller running on a Compute nodes uses OVN 2.11 while the southbound database on Controller nodes uses OVN 2.13. This causes issues when perform Compute-based operations, such as workload migration, during the upgrade. Red Hat aims to resolve this issue in the next 16.1 minor release update.

BZ#1882400 - New VMs with SR-IOV ports cannot be created

+ --- Due to changes with port binding, you cannot create a new VM with SR-IOV ports while Compute nodes are in a hybrid state. Create VMs with SR-IOV ports before you begin the pgrade or after you successfully complete the upgrade. For more information, see "During Framework Upgrade from OpenStack 13 to 16.1 new instances with SR-IOV ports cannot be created ". ---

Red Hat Ceph Storage Issues

BZ#1855813 - Ceph tools repository should be switched from RHCS3 to RHCS4 only after converge, before running external-upgrade
The ceph-ansible playbook collection on the undercloud deploys Red Hat Ceph Storage containers on the overcloud. To upgrade your environment, you must have Red Hat Ceph Storage 3 version of ceph-ansible to maintain Ceph Storage 3 containers through the upgrade. This guide includes instructions on how to retain ceph-ansible version 3 over the course of the upgrade until you are ready to upgrade to Ceph Storage 4. Before performing the 13 to 16.1 upgrade, you must perform a minor version update of your Red Hat OpenStack Platform 13 environment and ensure you have ceph-ansible version 3.2.46 or later.
BZ#1870617 - noout ceph flag not unset during the FFU workflow
This guide includes commands to set noout and norebalance flags before the Ceph Storage OSD cluster upgrade and unset them after you complete the Ceph Storage OSD cluster upgrade. Red Hat aims to automate this procedure in the next 16.1 minor release update.

Red Hat Enterprise Linux Issues

BZ#1861977 - RHSA-2020:3216 grub2 security update renders system unbootable
Applying the RHSA-2020:3216 security update stops the grub menu from loading. This issue occurs when using the shim-x64-15-14.el8_2 package. The shim-x64-15-15.el8_2 package and later versions of this package corrects this issue. If the grub menu does not appear when rebooting nodes, see the "System hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 or RHSA-2020:3217" solution.

2.8. Backup and restore

Before you upgrade your Red Hat OpenStack Platform 13 environment, back up the undercloud and overcloud control plane. For more information about backing up nodes with the Relax-and-recover (ReaR) utility, see the Undercloud and Control Plane Back Up and Restore guide.

2.9. Minor version update

Before you upgrade your Red Hat OpenStack Platform environment, update the environment to the latest minor version of your current release. For example, perform an update of your Red Hat OpenStack Platform 13 environment to the latest 13 before running the upgrade to Red Hat OpenStack Platform 16.1

For instructions on performing a minor version update for Red Hat OpenStack Platform 13, see "Keeping Red Hat OpenStack Platform Updated".

2.10. Proxy configuration

If you use a proxy with your Red Hat OpenStack Platform 13 environment, the proxy configuration in the /etc/environment file will persist past the operating system upgrade and the Red Hat OpenStack Platform 16.1 upgrade.

2.11. Validating Red Hat OpenStack Platform 13 before the upgrade

Before you upgrade to Red Hat OpenStack Platform 16.1, validate your undercloud and overcloud with the tripleo-validations playbooks. In Red Hat OpenStack Platform 13, you run these playbooks through the OpenStack Workflow Service (mistral).

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Create a bash script called pre-upgrade-validations.sh and include the following content in the script:

    #!/bin/bash
    for VALIDATION in $(openstack action execution run tripleo.validations.list_validations '{"groups": ["pre-upgrade"]}' | jq ".result[] | .id")
    do
      echo "=== Running validation: $VALIDATION ==="
      STACK_NAME=$(openstack stack list -f value -c 'Stack Name')
      ID=$(openstack workflow execution create -f value -c ID tripleo.validations.v1.run_validation "{\"validation_name\": $VALIDATION, \"plan\": \"$STACK_NAME\"}")
      while [ $(openstack workflow execution show $ID -f value -c State) == "RUNNING" ]
      do
        sleep 1
      done
      echo ""
      openstack workflow execution output show $ID | jq -r ".stdout"
      echo ""
    done
  4. Add permission to run the script:

    $ chmod +x pre-upgrade-validations.sh
  5. Run the script:

    $ ./pre-upgrade-validations.sh

    Review the script output to determine which validations succeed and fail:

    === Running validation: "check-ftype" ===
    
    Success! The validation passed for all hosts:
    * undercloud