Chapter 2. Planning and preparation for an in-place upgrade

Before you conduct an in-place upgrade of your OpenStack Platform environment, create a plan for the upgrade and accommodate any potential obstacles that might block a successful upgrade.

2.1. Familiarize yourself with Red Hat OpenStack Platform 16.1

Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 16.1 to help you understand the resulting environment and any potential version-to-version changes what might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 16.1, follow these suggestions:

2.2. High level changes in Red Hat OpenStack Platform 16.1

The following high-level changes occur during the upgrade to Red Hat OpenStack Platform 16.1:

  • OpenStack Platform director 16.1 configures the overcloud using an Ansible-driven method called config-download. This replaces the standard heat-based configuration method. Director still uses heat to orchestrate provisioning operations.
  • The director installation uses the same method as the overcloud deployment. Therefore, the undercloud also uses openstack-tripleo-heat-templates as a blueprint for installing and configuring each service.
  • The undercloud runs OpenStack services in containers.
  • The undercloud pulls and stores container images through a new method. Instead of pulling container images before deploying the overcloud, the undercloud pulls all relevant container images during the deployment process.
  • The overcloud deployment process includes an Advanced Subscription Management method to register nodes. This method incorporates an Ansible role to register OpenStack Platform nodes. The new method also applies different subscriptions to different node roles if necessary.
  • The overcloud now uses Open Virtual Network (OVN) as the default ML2 mechanism driver. It is possible to migrate your Open vSwitch (OVS) service to OVN, which you perform after the completion of a successful upgrade.
  • The undercloud and overcloud both run on Red Hat Enterprise Linux 8.
  • openstack-tripleo-heat-templates includes a unified composable service template collection in the deployment directory. This directory now includes templates with merged content from both the containerized service and Puppet-based composable service templates.
  • The OpenStack Data Processing service (sahara) is no longer supported.

    Important

    If you have sahara enabled in your Red Hat OpenStack Platform 13 environment, do not continue with this upgrade and contact Red Hat Global Support Services.

  • The OpenStack Telemetry components are deprecated in favor of the Service Telemetry Framework (STF).

2.3. Changes in Red Hat Enterprise Linux 8

The undercloud and overcloud both run on Red Hat Enterprise Linux 8. This includes new tools and functions relevant to the undercloud and overcloud:

  • The undercloud and overcloud use the Red Hat Container Toolkit. Instead of docker to build and control the container lifecycle, Red Hat Enterprise Linux 8 includes buildah to build new container images and podman for container management.
  • Red Hat Enterprise Linux 8 does not include the docker-distribution package. The undercloud now includes a private HTTP registry to provide container images to overcloud nodes.
  • The upgrade process from Red Hat Enterprise Linux 7 to 8 uses the leapp tool.
  • Red Hat Enterprise Linux 8 does not use the ntp service. Instead, Red Hat Enterprise Linux 8 uses chronyd.
  • Red Hat Enterprise Linux 8 includes new versions of high availability tools.

The Red Hat OpenStack Platform 16.1 uses Red Hat Enterprise Linux 8.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to Red Hat Enterprise Linux 8.2.

For more information about the key differences between Red Hat Enterprise Linux 7 and 8, see "Considerations in adopting RHEL 8". For general information about Red Hat Enterprise linux 8, see Product Documentation for Red Hat Enterprise Linux 8.

2.4. Leapp upgrade usage in Red Hat OpenStack Platform

The long-life Red Hat OpenStack Platform upgrade requires a base operating system upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8. Red Hat Enterprise Linux 7 uses the Leapp utility to perform the upgrade to Red Hat Enterprise Linux 8. To ensure that Leapp and its dependencies are available, verify that the following Red Hat Enterprise Linux 7 repositories are enabled:

  • Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server or Red Hat Enterprise Linux 7 Server RPMs x86_64 7.9

    rhel-7-server-rpms
    x86_64 7Server
    or:
    rhel-7-server-rpms
    x86_64 7.9
  • Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64

    rhel-7-server-extras-rpms
    x86_64

For more information, see Preparing a RHEL 7 system for the upgrade.

The undercloud and overcloud use a separate process for performing the operating system upgrade.

Undercloud process

Run the leapp upgrade manually before you run the openstack undercloud upgrade command. The undercloud upgrade includes instructions for performing the leapp upgrade.

Overcloud process

The overcloud upgrade framework automatically runs the leapp upgrade.

Limitations

For information of potential limitations that might affect your upgrade, see the following sections from the Upgrading from RHEL 7 to RHEL 8 guide:

In particular, you cannot perform a Leapp upgrade on nodes that use encryption of the whole disk or a partition, such as LUKS encryption, or file-system encryption. This limitation affects Ceph OSD nodes that you have configured with the dmcrypt: true parameter.

If any known limitations affect your environment, seek advice from the Red Hat Technical Support Team.

Troubleshooting

For information about troubleshooting potential Leapp issues, see Troubleshooting in Upgrading from RHEL 7 to RHEL 8.

2.5. Supported upgrade scenarios

Before proceeding with the upgrade, check that your overcloud is supported.

Note

If you are uncertain whether a particular scenario not mentioned in these lists is supported, seek advice from the Red Hat Technical Support Team.

Supported scenarios

The following in-place upgrade scenarios are tested and supported.

  • Standard environments with default role types: Controller, Compute, and Ceph Storage OSD
  • Split-Controller composable roles
  • Ceph Storage composable roles
  • Hyper-Converged Infrastructure: Compute and Ceph Storage OSD services on the same node
  • Environments with Network Functions Virtualization (NFV) technologies: Single-root input/output virtualization (SR-IOV) and Data Plane Development Kit (DPDK)
  • Environments with Instance HA enabled

Technology preview scenarios

The framework for upgrades is considered a Technology Preview when you use it in conjunction with these features, and therefore is not fully supported by Red Hat. You should only test this scenario in a proof-of-concept environment and not upgrade in a production environment. For more information about Technology Preview features, see Scope of Coverage Details.

  • Edge and Distributed Compute Node (DCN) scenarios

2.6. Considerations for upgrading with external Ceph deployments

If you have deployed a Red Hat Ceph Storage system separately and then used director to deploy and configure OpenStack, you can use the Red Hat OpenStack Platform framework for upgrades to perform an in-place upgrade with external Ceph deployments. This scenario is different from upgrading a Ceph cluster that was deployed using director.

The differences that you must take into account when planning and preparing for an in-place upgrade with external Ceph deployments are the following:

  1. Before you can upgrade your Red Hat OpenStack Platform deployment from version 13 to version 16.1, you must upgrade your Red Hat Ceph Storage cluster from version 3 to version 4. For more information, see Upgrading a Red Hat Ceph Storage cluster in the Red Hat Ceph Storage 4 Installation Guide.
  2. After you upgrade your Red Hat Ceph Storage cluster from version 3 to version 4, Red Hat OpenStack Platform 13 might still run RHCSv3 client components, however these are compatible against the RHCSv4 cluster.
  3. You can follow the upgrade path described in the Framework For Upgrades (13 to 16.1) document, and where applicable, you must complete the conditional steps that support this particular scenario. A conditional step starts with the following statement: "If you are upgrading with external Ceph deployments".
  4. When you upgrade with external Ceph deployments, you install RHCSv4 ceph-ansible as part of the overcloud upgrade process. When you upgrade a Ceph cluster that was deployed using director, you install RHCSv4 ceph-ansible after the overcloud upgrade process is complete.
Important

When you upgrade a Red Hat Ceph Storage cluster from a previous supported version to version 4.2z2, the upgrade completes with the storage cluster in a HEALTH_WARN state with a warning message that states monitors are allowing insecure global_id reclaim. This is due to the patched CVE (CVE-2021-20288), see Ceph HEALTH_WARN with 'mons are allowing insecure global_id reclaim' after install/upgrade to RHCS 4.2z2 (or newer).

Because the HEALTH_WARN state is displayed due to the CVE, it is possible to mute health warnings temporarily. However, there is a risk that if you mute warnings you do not have visibility about potential older and unpatched clients connected to your cluster. For more information about muting health warnings, see Upgrading a Red Hat Ceph Storage cluster in the Red Hat Ceph Storage documentation.

Important

Do not disable global_id_reclaim manually until all clients are upgraded and patched otherwise they cannot connect. You can run the following command as the root user to view a list of unpatched clients that are connected to the cluster:

# ceph health detail

2.7. Known issues that might block an upgrade

Review the following known issues that might affect a successful upgrade.

BZ#1902849 - osp13-osp16.1 ffu fails on clusters previously upgraded from osp8, osp10
Red Hat OpenStack Platform (RHOSP) environments that have been previously upgraded from version RHOSP 10, require the python-docker package to avoid BZ#1902849. For more information, see the Red Hat Knowledgebase solution osp13-osp16.1 ffu fails on older environments missing python-docker package.
BZ#1925078 - RHOSP13-16.1 FFU: Overcloud upgrade hangs in controller after failed attempt with reference to wrong ceph image

Systems that use UEFI boot and a UEFI bootloader in OSP13 might run into an UEFI issue that results in:

  • /etc/fstab not being updated
  • grub-install is incorrectly used on EFI system

For more information, see the Red Hat Knowledgebase solution FFU 13 to 16.1: Leapp fails to update the kernel on UEFI based systems and /etc/fstab does not contain the EFI partition.

If your systems use UEFI, contact Red Hat Technical Support.

BZ#1895887 - ovs+dpdk fail to attach device OvsDpdkHCI

After upgrading with the Leapp utility, the Compute node with OVS-DPDK workload does not function properly. To resolve this issue, perform one of the following steps:

Remove the /etc/modules-load.d/vfio-pci.conf file before you upgrade the Compute node.

or

Restart ovs-vswitchd service on the Compute node after you upgrade the Compute node.

This issue affects RHOSP 16.1.3. For more information, see the Red Hat Knowledgebase solution OVS-DPDK errors after Framework Upgrade from OSP 13 to 16.1 on HCI compute node.

BZ#1936419 - FFU 13-16.1 Upgrade: Leapp upgrade on ceph nodes failing as leap parameters try to enable the Fast datapath repo

If you use a Ceph subscription and have configured director to use the overcloud-minimal image for Ceph storage nodes, the upgrade of the operating system for Ceph storage nodes might fail due to a Leapp limitation. To avoid this issue, after the system_upgrade run step, you must log in to the Ceph node to unset the RHEL minor release version, update to the latest available RHEL minor release version, and reboot the node.

If you use Red Hat Satellite Server to host RPM content for the Leapp upgrade, you must add the following 8.2 repositories to the Content View that you use:

  • Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)

    rhel-8-for-x86_64-appstream-rpms
    x86_64 8.2
  • Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)

    rhel-8-for-x86_64-baseos-rpms
    x86_64 8.2

    This guide includes a workaround to avoid this issue.

Red Hat Ceph Storage Issues

BZ#1855813 - Ceph tools repository should be switched from RHCS3 to RHCS4 only after converge, before running external-upgrade
The ceph-ansible playbook collection on the undercloud deploys Red Hat Ceph Storage containers on the overcloud. To upgrade your environment, you must have Red Hat Ceph Storage 3 version of ceph-ansible to maintain Ceph Storage 3 containers through the upgrade. This guide includes instructions on how to retain ceph-ansible version 3 over the course of the upgrade until you are ready to upgrade to Ceph Storage 4. Before performing the 13 to 16.1 upgrade, you must perform a minor version update of your Red Hat OpenStack Platform 13 environment and ensure you have ceph-ansible version 3.2.46 or later.

2.8. Backup and restore

Before you upgrade your Red Hat OpenStack Platform 13 environment, back up the undercloud and overcloud control plane. For more information about backing up nodes with the Relax-and-recover (ReaR) utility, see the Undercloud and Control Plane Back Up and Restore guide.

2.9. Minor version update

Before you upgrade your Red Hat OpenStack Platform environment, update the environment to the latest minor version of your current release. For example, perform an update of your Red Hat OpenStack Platform 13 environment to the latest 13 before running the upgrade to Red Hat OpenStack Platform 16.1

For instructions on performing a minor version update for Red Hat OpenStack Platform 13, see "Keeping Red Hat OpenStack Platform Updated".

2.10. Proxy configuration

If you use a proxy with your Red Hat OpenStack Platform 13 environment, the proxy configuration in the /etc/environment file will persist past the operating system upgrade and the Red Hat OpenStack Platform 16.1 upgrade.

2.11. Validating Red Hat OpenStack Platform 13 before the upgrade

Before you upgrade to Red Hat OpenStack Platform 16.1, validate your undercloud and overcloud with the tripleo-validations playbooks. In Red Hat OpenStack Platform 13, you run these playbooks through the OpenStack Workflow Service (mistral).

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Create a bash script called pre-upgrade-validations.sh and include the following content in the script:

    #!/bin/bash
    for VALIDATION in $(openstack action execution run tripleo.validations.list_validations '{"groups": ["pre-upgrade"]}' | jq ".result[] | .id")
    do
      echo "=== Running validation: $VALIDATION ==="
      STACK_NAME=$(openstack stack list -f value -c 'Stack Name')
      ID=$(openstack workflow execution create -f value -c ID tripleo.validations.v1.run_validation "{\"validation_name\": $VALIDATION, \"plan\": \"$STACK_NAME\"}")
      while [ $(openstack workflow execution show $ID -f value -c State) == "RUNNING" ]
      do
        sleep 1
      done
      echo ""
      openstack workflow execution output show $ID | jq -r ".stdout"
      echo ""
    done
  4. Add permission to run the script:

    $ chmod +x pre-upgrade-validations.sh
  5. Run the script:

    $ ./pre-upgrade-validations.sh

    Review the script output to determine which validations succeed and fail:

    === Running validation: "check-ftype" ===
    
    Success! The validation passed for all hosts:
    * undercloud