Framework for upgrades (16.2 to 17.1)
In-place upgrades from Red Hat OpenStack Platform 16.2 to 17.1
OpenStack Documentation Team
rhos-docs@redhat.com
Abstract
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your input on our documentation. Tell us how we can make it better.
Using the Direct Documentation Feedback (DDF) function
Use the Add Feedback DDF function for direct comments on specific sentences, paragraphs, or code blocks.
- View the documentation in the Multi-page HTML format.
- Ensure that you see the Feedback button in the upper right corner of the document.
- Highlight the part of text that you want to comment on.
- Click Add Feedback.
- Complete the Add Feedback field with your comments.
- Optional: Add your email address so that the documentation team can contact you for clarification on your issue.
- Click Submit.
Chapter 1. About the Red Hat OpenStack Platform framework for upgrades
The Red Hat OpenStack Platform (RHOSP) framework for upgrades is a workflow to upgrade your RHOSP environment from one long life version to the next long life version. This workflow is an in-place solution and the upgrade occurs within your existing environment.
1.1. High-level changes in Red Hat OpenStack Platform 17.1
The following high-level changes occur during the upgrade to Red Hat OpenStack Platform (RHOSP) 17.1:
- The RHOSP upgrade and the operating system upgrade are separated into two distinct phases. You upgrade RHOSP first, then you upgrade the operating system.
- You can upgrade a portion of your Compute nodes to RHEL 9.2 while the rest of your Compute nodes remain on RHEL 8.4. This is called a Multi-RHEL environment.
-
With an upgrade to Red Hat Ceph Storage 5,
cephadm
now manages Red Hat Ceph Storage. Previous versions of Red Hat Ceph Storage were managed byceph-ansible
. Red Hat Ceph Storage nodes can remain on version 5 with RHOSP 17.1 until the end of the Red Hat Ceph Storage 5 lifecycle. - The overcloud uses Open Virtual Network (OVN) as the default ML2 mechanism driver. It is possible to migrate your Open vSwitch (OVS) service to OVN, which you perform after completing a successful upgrade.
- The undercloud and overcloud both run on Red Hat Enterprise Linux 9.
1.2. Changes in Red Hat Enterprise Linux 9
The Red Hat OpenStack Platform 17.1 uses Red Hat Enterprise Linux (RHEL) 9.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to RHEL 9.2.
Before beginning the upgrade, review the following information to familiarize yourself with RHEL 9:
- For more information about the latest changes in RHEL 9, see the Red Hat Enterprise Linux 9.2 Release Notes.
- For more information about the key differences between Red Hat Enterprise Linux 8 and 9, see Considerations in adopting RHEL 9.
- For general information about Red Hat Enterprise Linux 9, see Product Documentation for Red Hat Enterprise Linux 9.
- For more information about upgrading from RHEL 8 to RHEL 9, see Upgrading from RHEL 8 to RHEL 9.
1.3. Upgrade framework for long life versions
You can use the Red Hat OpenStack Platform (RHOSP) upgrade framework to perform an in-place upgrade path through multiple versions of the overcloud. The goal is to provide you with an opportunity to remain on certain OpenStack versions that are considered long life versions and upgrade when the next long life version is available.
The Red Hat OpenStack Platform upgrade process also upgrades the version of Red Hat Enterprise Linux (RHEL) on your nodes.
This guide provides an upgrade framework through the following versions:
Current Version | Target Version |
---|---|
Red Hat OpenStack Platform 16.2.4 and later | Red Hat OpenStack Platform 17.1 latest |
For detailed support dates and information on the lifecycle support for Red Hat OpenStack Platform, see Red Hat OpenStack Platform Life Cycle.
Upgrade paths for long life releases
Familiarize yourself with the possible update and upgrade paths before you begin an upgrade. If you are using an environment that is earlier than RHOSP 16.2.4, before you upgrade from major version to major version, you must first update your existing environment to the latest minor release.
For example, if your current deployment is Red Hat OpenStack Platform (RHOSP) 16.2.1 on Red Hat Enterprise Linux (RHEL) 8.4, you must perform a minor update to the latest RHOSP 16.2 version before you upgrade to RHOSP 17.1.
You can view your current RHOSP and RHEL versions in the /etc/rhosp-release
and /etc/redhat-release
files.
Table 1.1. Updates version path
Current version | Target version |
RHOSP 16.2.x on RHEL 8.4 | RHOSP 16.2 latest on RHEL 8.4 latest |
RHOSP 17.0.x on RHEL 9.0 | RHOSP 17.0 latest on RHEL 9.0 latest |
RHOSP 17.0.x on RHEL 9.0 | RHOSP 17.1 latest on RHEL 9.2 latest |
RHOSP 17.1.x on RHEL 9.2 | RHOSP 17.1 latest on RHEL 9.2 latest |
For more information, see Performing a minor update of Red Hat OpenStack Platform.
Table 1.2. Upgrades version path
Current version | Target version |
RHOSP 16.2 on RHEL 8.4 | RHOSP 17.1 latest on RHEL 9.2 latest |
Red Hat provides two options for upgrading your environment to the next long life release:
- In-place upgrade
- Perform an upgrade of the services in your existing environment. This guide primarily focuses on this option.
- Parallel migration
- Create a new RHOSP 17.1 environment and migrate your workloads from your current environment to the new environment. For more information about RHOSP parallel migration, contact Red Hat Global Professional Services.
The durations in this table are minimal estimates based on internal testing and might not apply to all production environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. To accurately gauge the upgrade duration for each task, perform these procedures in a test environment with hardware similar to your production environment.
Table 1.3. Duration and impact of In-place upgrade
Duration | Impact | |
---|---|---|
Undercloud upgrade | 30 minutes | None |
Overcloud adoption | 6 minutes | None |
Overcloud upgrade preparation | 20 minutes | None |
Red Hat Ceph Storage upgrade | 30 minutes | None |
Red Hat Ceph Storage adoption | 30 minutes | None |
Overcloud OpenStack upgrade | 70 minutes | None |
Overcloud system upgrade | 30 minutes per node or can be done in parallel | None |
Overcloud compute upgrade | 20 minutes | None |
Overcloud outage | The outage is minimal due to workload migration from node to node. |
1.4. Planning and preparation for an in-place upgrade
Before you conduct an in-place upgrade of your OpenStack Platform environment, create a plan for the upgrade and accommodate any potential obstacles that might block a successful upgrade.
1.4.1. Familiarize yourself with Red Hat OpenStack Platform 17.1
Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 17.1 to help you understand the resulting environment and any potential version-to-version changes that might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 17.1, follow these suggestions:
Read the release notes for all versions across the upgrade path and identify any potential aspects that require planning:
- Components that contain new features
- Known issues
Open the release notes for each version using these links:
- Red Hat OpenStack Platform 16.2, which is your source version
- Red Hat OpenStack Platform 17.1 which is your target version
- Read the Installing and managing Red Hat OpenStack Platform with director guide for version 17.1 and familiarize yourself with any new requirements and processes in this guide.
- Install a proof-of-concept Red Hat OpenStack Platform 17.1 undercloud and overcloud. Develop hands-on experience of the target OpenStack Platform version and investigate potential differences between the target version and your current version.
1.4.2. Leapp upgrade usage in Red Hat OpenStack Platform
The long-life Red Hat OpenStack Platform upgrade requires a base operating system upgrade from Red Hat Enterprise Linux (RHEL) 8.4 to RHEL 9.2. The upgrade process uses the Leapp utility to perform the upgrade to RHEL 9.2. However, some aspects of the Leapp upgrade are customized to ensure that you are upgrading specifically from RHEL 8.4 to RHEL 9.2. To upgrade your operating system to RHEL 9.2, see Performing the undercloud system upgrade.
Limitations
For information on potential limitations that might affect your upgrade, see the following sections from the Upgrading from RHEL 8 to RHEL 9 guide:
If any known limitations affect your environment, seek advice from the Red Hat Technical Support Team.
Troubleshooting
For information about troubleshooting potential Leapp issues, see Troubleshooting in Upgrading from RHEL 8 to RHEL 9.
1.4.3. Supported upgrade scenarios
Before proceeding with the upgrade, check that your overcloud is supported.
If you are uncertain whether a particular scenario not mentioned in these lists is supported, seek advice from the Red Hat Technical Support Team.
Supported scenarios
The following in-place upgrade scenarios are tested and supported.
- Standard environments with default role types: Controller, Compute, and Ceph Storage OSD
- Split-Controller composable roles
- Ceph Storage composable roles
- Hyper-Converged Infrastructure: Compute and Ceph Storage OSD services on the same node
- Environments with Network Functions Virtualization (NFV) technologies: Single-root input/output virtualization (SR-IOV) and Data Plane Development Kit (DPDK)
Environments with Instance HA enabled
NoteDuring an upgrade procedure, nova live migrations are supported. However, evacuations initiated by Instance HA are not supported. When you upgrade a Compute node, the node is shut down cleanly and any workload running on the node is not evacuated by Instance HA automatically. Instead, you must perform live migration manually.
Technology preview scenarios
The framework for upgrades is considered a Technology Preview when you use it in conjunction with these features, and therefore is not fully supported by Red Hat. You should only test this scenario in a proof-of-concept environment and not upgrade in a production environment. For more information about Technology Preview features, see Scope of Coverage Details.
- Edge and Distributed Compute Node (DCN) scenarios
1.4.4. Considerations for upgrading with external Ceph deployments
If you have deployed a Red Hat Ceph Storage system separately and then used director to deploy and configure OpenStack, before you can upgrade your Red Hat OpenStack Platform (RHOSP) deployment from version 16.2 to version 17.1, you must upgrade your Red Hat Ceph Storage cluster from version 4 to version 5. After you upgrade your Red Hat Ceph Storage cluster, Red Hat OpenStack Platform 16 continues to operate and be compatible with the storage cluster. For more information, see Upgrading a Red Hat Ceph Storage cluster in the Red Hat Ceph Storage 5 Upgrade Guide.
1.4.5. Known issues that might block an upgrade
Review the following known issues that might affect a successful upgrade.
If you upgrade your operating system from RHEL 7.x to RHEL 8.x, or from RHEL 8.x to RHEL 9.x, do not run a Leapp upgrade with the --debug
option. The system remains in the early console in setup code
state and does not reboot automatically. To avoid this issue, the UpgradeLeappDebug
parameter is set to false
by default. Do not change this value in your templates.
After rebooting an overcloud node, a permission issue causes collectd-sensubility to stop working. As a result, collectd-sensubility stops reporting container health. During an upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, overcloud nodes are rebooted as part of the Leapp upgrade. To ensure that collectd-sensubility continues to work, run the following command:
sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman
The following deployments are not supported in a Multi-RHEL environment:
- ShiftOnStack
- DirectorOperator
You must upgrade all Compute hosts to RHEL 9.2, or keep all Compute hosts running on RHEL 8.4.
Distributed Compute Node (DCN) deployments do not support a Multi-RHEL environment, nor an upgrade from RHOSP 16.2 to RHOSP 17.1.
The Pacemaker-controlled ceph-nfs
resource requires a runtime directory to store some process data. The directory is created when you install or upgrade RHOSP. Currently, a reboot of the Controller nodes removes the directory, and the ceph-nfs
service does not recover when the Controller nodes are rebooted. If all Controller nodes are rebooted, the ceph-nfs
service fails permanently.
You can apply the following workaround:
If you reboot a Controller node, log in to the Controller node and create a
/var/run/ceph
directory:$ mkdir -p /var/run/ceph
Repeat this step on all Controller nodes that have been rebooted. If the
ceph-nfs-pacemaker
service has been marked as failed, after creating the directory, run the following command from any of the Controller nodes:$ pcs resource cleanup
After a Compute node upgrade from RHEL 8.4 to RHEL 9.2, the nova_virtlogd
container image continues to run. This container image uses Red Hat Universal Base Image (UBI) 8 containers. Additionally, another container image called nova_virtlogd_wrapper
is created, which uses UBI 9 containers. All container images should use UBI 9 containers after the upgrade. However, the nova_virtlogd
container image does not affect the environment. A fix is expected in a later RHOSP release.
The RHOSP upgrade from 16.2 to 17.1 fails when pulling images from registry.redhat.io
because the upgrade playbook does not include the podman registry login task. Contact your Red Hat support representative for a hotfix. A fix is expected in a later RHOSP release.
If your node has GlanceApi
NFS mounts and you are upgrading RHOSP from 16.2 to 17.1, the NFS mounts are removed before the operating system upgrade and not restored afterward. If your node has Cinder-Volume
or Nova-Compute
NFS mounts, they are not removed and the operating system upgrade does not run. The Red Hat engineering team is investigating a solution to this issue.
1.4.6. Backup and restore
Before you upgrade your Red Hat OpenStack Platform (RHOSP) 16.2 environment, back up the undercloud and overcloud control plane by using one of the following options:
- Back up your nodes before you perform an upgrade. For more information about backing up nodes before you upgrade, see Red Hat OpenStack Platform 16.2 Backing up and restoring the undercloud and control plane nodes.
- Back up the undercloud node after you perform the undercloud upgrade and before you perform the overcloud upgrade. For more information about backing up the undercloud, see Creating a backup of the undercloud node in the Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes.
- Use a third-party backup and recovery tool that suits your environment. For more information about certified backup and recovery tools, see the Red Hat Ecosystem catalog.
1.4.7. Minor version update
If you are using a version of Red Hat OpenStack Platform (RHOSP) that is earlier than 16.2.4, before you upgrade your RHOSP environment, update the environment to the latest minor version of your current release. For example, update your Red Hat OpenStack Platform 16.2.3 environment to the latest 16.2 version before upgrading to Red Hat OpenStack Platform 17.1.
For instructions on performing a minor version update for Red Hat OpenStack Platform 16.2, see Keeping Red Hat OpenStack Platform Updated.
1.4.8. Proxy configuration
If you use a proxy with your Red Hat OpenStack Platform 16.2 environment, the proxy configuration in the /etc/environment
file will persist past the operating system upgrade and the Red Hat OpenStack Platform 17.1 upgrade.
- For more information about proxy configuration for Red Hat OpenStack Platform 16.2, see Considerations when running the undercloud with a proxy in Installing and managing Red Hat OpenStack Platform with director.
- For more information about proxy configuration for Red Hat OpenStack Platform 17.1, see Considerations when running the undercloud with a proxy in Installing and managing Red Hat OpenStack Platform with director.
1.4.9. Planning for a Compute node upgrade
After you upgrade your Compute nodes from Red Hat OpenStack Platform (RHOSP) 16.2 to RHOSP 17.1, you can choose one of the following options to upgrade the Compute host operating system:
- Keep a portion of your Compute nodes on Red Hat Enterprise Linux (RHEL) 8.4, and upgrade the rest to RHEL 9.2. This is referred to as a Multi-RHEL environment.
- Upgrade all Compute nodes to RHEL 9.2, and complete the upgrade of the environment.
- Keep all Compute nodes on RHEL 8.4. The lifecycle of RHEL 8.4 applies.
Benefits of a Multi-RHEL environment
You must upgrade all of your Compute nodes to RHEL 9.2 to take advantage of any hardware-related features that are only supported in RHOSP 17.1, such as vTPM and Secure Boot. However, you might require that some or all of your Compute nodes remain on RHEL 8.4. For example, if you certified an application for RHEL 8, you can keep your Compute nodes running on RHEL 8.4 to support the application without blocking the entire upgrade.
The option to upgrade part of your Compute nodes to RHEL 9.2 gives you more control over your upgrade process. You can prioritize upgrading the RHOSP environment within a planned maintenance window and defer the operating system upgrade to another time. Less downtime is required, which minimizes the impact to end users.
If you plan to upgrade from RHOSP 17.1 to RHOSP 18.0, you must upgrade all hosts to RHEL 9.2. If you continue to run RHEL 8.4 on your hosts beyond the Extended Life Cycle Support phase, you must obtain a TUS subscription.
Limitations of a Multi-RHEL environment
The following limitations apply in a Multi-RHEL environment:
- Compute nodes running RHEL 8 cannot consume NVMe-over-TCP Cinder volumes.
- You cannot use different paths for socket files on RHOSP 16.2 and 17.1 for collectd monitoring.
- You cannot mix ML2/OVN and ML2/OVS mechanism drivers. For example, if your RHOSP 16.2 deployment included ML2/OVN, your Multi-RHEL environment must use ML2/OVN.
- FIPS is not supported in a Multi-RHEL environment. FIPs deployment is a Day 1 operation. FIPS is not supported in RHOSP 16.2. As a result, when you upgrade from RHOSP 16.2 to 17.1, the 17.1 environment does not include FIPS.
- Edge topologies are currently not supported.
All HCI nodes in supported Hyperconverged Infrastructure environments must use the same version of Red Hat Enterprise Linux as the version used by the Red Hat OpenStack Platform controllers. If you wish to use multiple Red Hat Enterprise versions in a hybrid state on HCI nodes in the same Hyperconverged Infrastructure environment, contact the Red Hat Customer Experience and Engagement team to discuss a support exception.
Upgrading Compute nodes
Use one of the following options to upgrade your Compute nodes:
- To perform a Multi-RHEL upgrade of your Compute nodes, see Upgrading Compute nodes to a Multi-RHEL environment.
- To upgrade all Compute nodes to RHEL 9.2, see Upgrading Compute nodes to RHEL 9.2.
- If you are keeping all of your Compute nodes on RHEL 8.4, no additional configuration is required.
1.5. Repositories
This section contains the repositories for the undercloud and overcloud. Refer to this section when you need to enable repositories in certain situations:
- Enabling repositories when registering to the Red Hat Customer Portal.
- Enabling and synchronizing repositories to your Red Hat Satellite Server.
- Enabling repositories when registering to your Red Hat Satellite Server.
1.5.1. Undercloud repositories
You run Red Hat OpenStack Platform (RHOSP) 17.1 on Red Hat Enterprise Linux (RHEL) 9.2. RHEL 8.4 Compute nodes are also supported in a Multi-RHEL environment when upgrading from RHOSP 16.2.
If you synchronize repositories with Red Hat Satellite, you can enable specific versions of the Red Hat Enterprise Linux repositories. However, the repository remains the same despite the version you choose. For example, you can enable the 9.2 version of the BaseOS repository, but the repository name is still rhel-9-for-x86_64-baseos-eus-rpms
despite the specific version you choose.
Any repositories except the ones specified here are not supported. Unless recommended, do not enable any other products or repositories except the ones listed in the following tables or else you might encounter package dependency issues. Do not enable Extra Packages for Enterprise Linux (EPEL).
Core repositories
The following table lists core repositories for installing the undercloud.
Name | Repository | Description of requirement |
---|---|---|
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) |
| High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability. |
Red Hat OpenStack Platform for RHEL 9 (RPMs) |
| Core Red Hat OpenStack Platform repository, which contains packages for Red Hat OpenStack Platform director. |
Red Hat Fast Datapath for RHEL 9 (RPMS) |
| Provides Open vSwitch (OVS) packages for OpenStack Platform. |
Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) |
| Core Red Hat OpenStack Platform repository. |
1.5.2. Overcloud repositories
You run Red Hat OpenStack Platform (RHOSP) 17.1 on Red Hat Enterprise Linux (RHEL) 9.2. RHEL 8.4 Compute nodes are also supported in a Multi-RHEL environment when upgrading from RHOSP 16.2.
If you synchronize repositories with Red Hat Satellite, you can enable specific versions of the Red Hat Enterprise Linux repositories. However, the repository remains the same despite the version you choose. For example, you can enable the 9.2 version of the BaseOS repository, but the repository name is still rhel-9-for-x86_64-baseos-eus-rpms
despite the specific version you choose.
Any repositories except the ones specified here are not supported. Unless recommended, do not enable any other products or repositories except the ones listed in the following tables or else you might encounter package dependency issues. Do not enable Extra Packages for Enterprise Linux (EPEL).
Controller node repositories
The following table lists core repositories for Controller nodes in the overcloud.
Name | Repository | Description of requirement |
---|---|---|
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) |
| High availability tools for Red Hat Enterprise Linux. |
Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) |
| Core Red Hat OpenStack Platform repository. |
Red Hat Fast Datapath for RHEL 9 (RPMS) |
| Provides Open vSwitch (OVS) packages for OpenStack Platform. |
Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) |
| Tools for Red Hat Ceph Storage 6 for Red Hat Enterprise Linux 9. |
Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) |
| Core Red Hat OpenStack Platform repository. |
Compute and ComputeHCI node repositories
The following table lists core repositories for Compute and ComputeHCI nodes in the overcloud.
Name | Repository | Description of requirement |
---|---|---|
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat Enterprise Linux 9 for x86_64 - High Availability (RPMs) Extended Update Support (EUS) |
| High availability tools for Red Hat Enterprise Linux. |
Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) |
| Core Red Hat OpenStack Platform repository. |
Red Hat Fast Datapath for RHEL 9 (RPMS) |
| Provides Open vSwitch (OVS) packages for OpenStack Platform. |
Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) |
| Tools for Red Hat Ceph Storage 6 for Red Hat Enterprise Linux 9. |
Red Hat Enterprise Linux 8.4 for x86_64 - BaseOS (RPMs) Telecommunications Update Service (TUS) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 8.4 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat OpenStack Platform for RHEL 8 x86_64 (RPMs) |
| Core Red Hat OpenStack Platform repository. |
Ceph Storage node repositories
The following table lists Ceph Storage related repositories for the overcloud.
Name | Repository | Description of requirement |
---|---|---|
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) |
| Base operating system repository for x86_64 systems. |
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) |
| Contains Red Hat OpenStack Platform dependencies. |
Red Hat OpenStack Platform Deployment Tools for RHEL 9 x86_64 (RPMs) |
|
Packages to help director configure Ceph Storage nodes. This repository is included with standalone Ceph Storage subscriptions. If you use a combined OpenStack Platform and Ceph Storage subscription, use the |
Red Hat OpenStack Platform for RHEL 9 x86_64 (RPMs) |
|
Packages to help director configure Ceph Storage nodes. This repository is included with combined Red Hat OpenStack Platform and Red Hat Ceph Storage subscriptions. If you use a standalone Red Hat Ceph Storage subscription, use the |
Red Hat Ceph Storage Tools 6 for RHEL 9 x86_64 (RPMs) |
| Provides tools for nodes to communicate with the Ceph Storage cluster. |
Red Hat Fast Datapath for RHEL 9 (RPMS) |
| Provides Open vSwitch (OVS) packages for OpenStack Platform. If you are using OVS on Ceph Storage nodes, add this repository to the network interface configuration (NIC) templates. |
1.5.3. Red Hat Satellite Server 6 considerations
If you use Red Hat Satellite Server 6 to host RPMs and container images for your Red Hat OpenStack Platform (RHOSP) environment and you plan to use Satellite 6 to deliver content during the RHOSP 17.1 upgrade, the following must be true:
- Your Satellite Server hosts RHOSP 16.2 RPMs and container images.
You have registered all nodes in your RHOSP 16.2 environment to your Satellite Server.
For example, you used an activation key linked to a RHOSP 16.2 content view to register nodes to RHOSP 16.2 content.
Recommendations for RHOSP upgrades
- Enable and synchronize the necessary RPM repositories for both the RHOSP 16.2 undercloud and overcloud. This includes the necessary Red Hat Enterprise Linux (RHEL) 9.2 repositories.
- Create custom products on your Satellite Server to host container images for RHOSP 17.1.
Create and promote a content view for RHOSP 17.1 upgrade and include the following content in the content view:
RHEL 8 repositories:
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-appstream-tus-rpms
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
rhel-8-for-x86_64-baseos-tus-rpms
Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs)
rhel-8-for-x86_64-highavailability-eus-rpms
Red Hat Fast Datapath for RHEL 8 (RPMs)
fast-datapath-for-rhel-8-x86_64-rpms
RHEL 9 repositories:
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-appstream-eus-rpms
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
rhel-9-for-x86_64-baseos-eus-rpms
- All undercloud and overcloud RPM repositories, including RHEL 9.2 repositories. To avoid issues enabling the RHEL repositories, ensure that you include the correct version of the RHEL repositories, which is 9.2.
- RHOSP 17.1 container images.
- Associate an activation key with the RHOSP 17.1 content view that you have created for the RHOSP 17.1 upgrade.
-
Check that no node has the
katello-host-tools-fact-plugin
package installed. The Leapp upgrade does not upgrade this package. Leaving this package on a RHEL 9.2 system causessubscription-manager
to report errors. - You can configure Satellite Server to host RHOSP 17.1 container images. For more information, see Preparing a Satellite server for container images in Installing and managing Red Hat OpenStack Platform with director.
If you use a Red Hat Ceph Storage subscription and have configured director to use the
overcloud-minimal
image for Red Hat Ceph Storage nodes, on your Satellite Server you must create a content view and add the following RHEL 9.2 repositories to it:Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-appstream-eus-rpms
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
rhel-9-for-x86_64-baseos-eus-rpms
For more information, see Importing Content and Managing Content Views in the Red Hat Satellite Managing Content guide.
Chapter 2. Upgrading the undercloud
Upgrade the undercloud to Red Hat OpenStack Platform 17.1. The undercloud upgrade uses the running Red Hat OpenStack Platform 16.2 undercloud. The upgrade process exports heat stacks to files, and converts heat to ephemeral heat while upgrading the rest of the services on your nodes.
2.1. Enabling repositories for the undercloud
Enable the repositories that are required for the undercloud, and update the system packages to the latest versions.
Procedure
-
Log in to your undercloud as the
stack
user. Disable all default repositories, and enable the required Red Hat Enterprise Linux (RHEL) repositories:
[stack@director ~]$ sudo subscription-manager repos --disable=* [stack@director ~]$ sudo subscription-manager repos \ --enable=rhel-8-for-x86_64-baseos-tus-rpms \ --enable=rhel-8-for-x86_64-appstream-tus-rpms \ --enable=rhel-8-for-x86_64-highavailability-tus-rpms \ --enable=openstack-17.1-for-rhel-8-x86_64-rpms \ --enable=fast-datapath-for-rhel-8-x86_64-rpms
Switch the
container-tools
module version to RHEL 8 on all nodes:[stack@director ~]$ sudo dnf -y module switch-to container-tools:rhel8
Install the command line tools for director installation and configuration:
[stack@director ~]$ sudo dnf install -y python3-tripleoclient
2.2. Validating RHOSP before the upgrade
Before you upgrade to Red Hat OpenStack Platform (RHOSP) 17.1, validate your undercloud and overcloud with the tripleo-validations
playbooks. In RHOSP 16.2, you run these playbooks through the OpenStack Workflow Service (mistral).
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Install the packages for validation:
$ sudo dnf -y update openstack-tripleo-validations python3-validations-libs validations-common
Copy the inventory from mistral:
$ sudo chown stack:stack /var/lib/mistral/.ssh/tripleo-admin-rsa $ sudo cat /var/lib/mistral/<stack>/tripleo-ansible-inventory.yaml > inventory.yaml
- Replace <stack> with the name of the stack.
Run the validation:
$ validation run -i inventory.yaml --group pre-upgrade
Review the script output to determine which validations succeed and fail:
=== Running validation: "check-ftype" === Success! The validation passed for all hosts: * undercloud
2.3. Preparing container images
The undercloud installation requires an environment file to determine where to obtain container images and how to store them. Generate and customize the environment file that you can use to prepare your container images.
If you need to configure specific container image versions for your undercloud, you must pin the images to a specific version. For more information, see Pinning container images for the undercloud.
Procedure
-
Log in to the undercloud host as the
stack
user. Optional: Back up the 16.2
containers-prepare-parameter.yaml
file:$ cp containers-prepare-parameter.yaml \ containers-prepare-parameter.yaml.orig
Generate the default container image preparation file:
$ openstack tripleo container image prepare default \ --local-push-destination \ --output-env-file containers-prepare-parameter.yaml
This command includes the following additional options:
-
--local-push-destination
sets the registry on the undercloud as the location for container images. This means that director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. Director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option. --output-env-file
is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file iscontainers-prepare-parameter.yaml
.NoteYou can use the same
containers-prepare-parameter.yaml
file to define a container image source for both the undercloud and the overcloud.
-
-
Modify the
containers-prepare-parameter.yaml
to suit your requirements. For more information about container image parameters, see Container image preparation parameters. If your deployment includes Red Hat Ceph Storage, update the Red Hat Ceph Storage container image parameters in the
containers-prepare-parameter.yaml
file for the version of Red Hat Ceph Storage that your deployment uses:ceph_namespace: registry.redhat.io/rhceph ceph_image: <ceph_image_file> ceph_tag: latest ceph_grafana_image: <grafana_image_file> ceph_grafana_namespace: registry.redhat.io/rhceph ceph_grafana_tag: latest
Replace
<ceph_image_file>
with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:-
Red Hat Ceph Storage 5:
rhceph-5-rhel8
-
Red Hat Ceph Storage 5:
Replace
<grafana_image_file>
with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:-
Red Hat Ceph Storage 5:
rhceph-5-dashboard-rhel8
-
Red Hat Ceph Storage 5:
2.4. Guidelines for container image tagging
The Red Hat Container Registry uses a specific version format to tag all Red Hat OpenStack Platform container images. This format follows the label metadata for each container, which is version-release
.
- version
- Corresponds to a major and minor version of Red Hat OpenStack Platform. These versions act as streams that contain one or more releases.
- release
- Corresponds to a release of a specific container image version within a version stream.
For example, if the latest version of Red Hat OpenStack Platform is 17.1.0 and the release for the container image is 5.161
, then the resulting tag for the container image is 17.1.0-5.161.
The Red Hat Container Registry also uses a set of major and minor version
tags that link to the latest release for that container image version. For example, both 17.1 and 17.1.0 link to the latest release
in the 17.1.0 container stream. If a new minor release of 17.1 occurs, the 17.1 tag links to the latest release
for the new minor release stream while the 17.1.0 tag continues to link to the latest release
within the 17.1.0 stream.
The ContainerImagePrepare
parameter contains two sub-parameters that you can use to determine which container image to download. These sub-parameters are the tag
parameter within the set
dictionary, and the tag_from_label
parameter. Use the following guidelines to determine whether to use tag
or tag_from_label
.
The default value for
tag
is the major version for your OpenStack Platform version. For this version it is 17.1. This always corresponds to the latest minor version and release.parameter_defaults: ContainerImagePrepare: - set: ... tag: 17.1 ...
To change to a specific minor version for OpenStack Platform container images, set the tag to a minor version. For example, to change to 17.1.2, set
tag
to 17.1.2.parameter_defaults: ContainerImagePrepare: - set: ... tag: 17.1.2 ...
-
When you set
tag
, director always downloads the latest container imagerelease
for the version set intag
during installation and updates. If you do not set
tag
, director uses the value oftag_from_label
in conjunction with the latest major version.parameter_defaults: ContainerImagePrepare: - set: ... # tag: 17.1 ... tag_from_label: '{version}-{release}'
The
tag_from_label
parameter generates the tag from the label metadata of the latest container image release it inspects from the Red Hat Container Registry. For example, the labels for a certain container might use the followingversion
andrelease
metadata:"Labels": { "release": "5.161", "version": "17.1.0", ... }
-
The default value for
tag_from_label
is{version}-{release}
, which corresponds to the version and release metadata labels for each container image. For example, if a container image has 17.1.0 set forversion
and 5.161 set forrelease
, the resulting tag for the container image is 17.1.0-5.161. -
The
tag
parameter always takes precedence over thetag_from_label
parameter. To usetag_from_label
, omit thetag
parameter from your container preparation configuration. -
A key difference between
tag
andtag_from_label
is that director usestag
to pull an image only based on major or minor version tags, which the Red Hat Container Registry links to the latest image release within a version stream, while director usestag_from_label
to perform a metadata inspection of each container image so that director generates a tag and pulls the corresponding image.
2.5. Obtaining container images from private registries
The registry.redhat.io
registry requires authentication to access and pull images. To authenticate with registry.redhat.io
and other private registries, include the ContainerImageRegistryCredentials
and ContainerImageRegistryLogin
parameters in your containers-prepare-parameter.yaml
file.
ContainerImageRegistryCredentials
Some container image registries require authentication to access images. In this situation, use the ContainerImageRegistryCredentials
parameter in your containers-prepare-parameter.yaml
environment file. The ContainerImageRegistryCredentials
parameter uses a set of keys based on the private registry URL. Each private registry URL uses its own key and value pair to define the username (key) and password (value). This provides a method to specify credentials for multiple private registries.
parameter_defaults: ContainerImagePrepare: - push_destination: true set: namespace: registry.redhat.io/... ... ContainerImageRegistryCredentials: registry.redhat.io: my_username: my_password
In the example, replace my_username
and my_password
with your authentication credentials. Instead of using your individual user credentials, Red Hat recommends creating a registry service account and using those credentials to access registry.redhat.io
content.
To specify authentication details for multiple registries, set multiple key-pair values for each registry in ContainerImageRegistryCredentials
:
parameter_defaults: ContainerImagePrepare: - push_destination: true set: namespace: registry.redhat.io/... ... - push_destination: true set: namespace: registry.internalsite.com/... ... ... ContainerImageRegistryCredentials: registry.redhat.io: myuser: 'p@55w0rd!' registry.internalsite.com: myuser2: '0th3rp@55w0rd!' '192.0.2.1:8787': myuser3: '@n0th3rp@55w0rd!'
The default ContainerImagePrepare
parameter pulls container images from registry.redhat.io
, which requires authentication.
For more information, see Red Hat Container Registry Authentication.
ContainerImageRegistryLogin
The ContainerImageRegistryLogin
parameter is used to control whether an overcloud node system needs to log in to the remote registry to fetch the container images. This situation occurs when you want the overcloud nodes to pull images directly, rather than use the undercloud to host images.
You must set ContainerImageRegistryLogin
to true
if push_destination
is set to false or not used for a given strategy.
parameter_defaults: ContainerImagePrepare: - push_destination: false set: namespace: registry.redhat.io/... ... ... ContainerImageRegistryCredentials: registry.redhat.io: myuser: 'p@55w0rd!' ContainerImageRegistryLogin: true
However, if the overcloud nodes do not have network connectivity to the registry hosts defined in ContainerImageRegistryCredentials
and you set ContainerImageRegistryLogin
to true
, the deployment might fail when trying to perform a login. If the overcloud nodes do not have network connectivity to the registry hosts defined in the ContainerImageRegistryCredentials
, set push_destination
to true
and ContainerImageRegistryLogin
to false
so that the overcloud nodes pull images from the undercloud.
parameter_defaults: ContainerImagePrepare: - push_destination: true set: namespace: registry.redhat.io/... ... ... ContainerImageRegistryCredentials: registry.redhat.io: myuser: 'p@55w0rd!' ContainerImageRegistryLogin: false
2.6. Updating the undercloud.conf file
You can continue using the original undercloud.conf
file from your Red Hat OpenStack Platform 16.2 environment, but you must modify the file to retain compatibility with Red Hat OpenStack Platform 17.1. For more information about parameters for configuring the undercloud.conf
file, see Director configuration parameters in Installing and managing Red Hat OpenStack Platform with director.
Procedure
-
Log in to your undercloud host as the
stack
user. Create a file called
skip_rhel_release.yaml
and set theSkipRhelEnforcement
parameter totrue
:parameter_defaults: SkipRhelEnforcement: true
Open the
undercloud.conf
file and add the following parameters to theDEFAULT
section in the file:container_images_file = /home/stack/containers-prepare-parameter.yaml custom_env_files = /home/stack/skip_rhel_release.yaml
Add any additional custom environment files to the
custom_env_files
parameter.The
custom_env_files
parameter defines the location of theskip_rhel_release.yaml
file that is required for the upgrade.-
The
container_images_file
parameter defines the location of thecontainers-prepare-parameter.yaml
environment file so that director pulls container images for the undercloud from the correct location.
- Check all other parameters in the file for any changes.
- Save the file.
2.7. Running the director upgrade
Upgrade director on the undercloud.
Procedure
Launch the director configuration script to upgrade director:
$ openstack undercloud upgrade
The director configuration script upgrades director packages and configures director services to match the settings in the
undercloud.conf
file. This script takes several minutes to complete.NoteThe director configuration script prompts for confirmation before proceeding. Bypass this confirmation by using the
-y
option:$ openstack undercloud upgrade -y
Chapter 3. Preparing for an overcloud upgrade
You must complete some initial steps to prepare for the overcloud upgrade.
3.1. Preparing for overcloud service downtime
The overcloud upgrade process disables the main control plane services at key points. You cannot use any overcloud services to create new resources when these key points are reached. Workloads that are running in the overcloud remain active during the upgrade process, which means instances continue to run during the upgrade of the control plane. During an upgrade of Compute nodes, these workloads can be live migrated to Compute nodes that are already upgraded.
It is important to plan a maintenance window to ensure that no users can access the overcloud services during the upgrade.
Affected by overcloud upgrade
- OpenStack Platform services
Unaffected by overcloud upgrade
- Instances running during the upgrade
- Ceph Storage OSDs (backend storage for instances)
- Linux networking
- Open vSwitch networking
- Undercloud
3.2. Disabling fencing in the overcloud
Before you upgrade the overcloud, ensure that fencing is disabled.
When you upgrade the overcloud, you upgrade each Controller node individually to retain high availability functionality. If fencing is deployed in your environment, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results.
If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the upgrade to avoid any unintended results.
When you complete the upgrade of your Red Hat OpenStack Platform environment, you must re-enable fencing in the overcloud. For more information about re-enabling fencing, see Re-enabling fencing in the overcloud.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Log in to a Controller node and run the Pacemaker command to disable fencing:
$ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"
-
Replace
<controller_ip>
with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with themetalsmith list
command.
-
Replace
-
In the
fencing.yaml
environment file, set theEnableFencing
parameter tofalse
to ensure that fencing stays disabled during the upgrade process.
Additional Resources
3.3. Undercloud node database backup
You can use the openstack undercloud backup
command to create a backup of the database that runs on the undercloud node and use that backup to recover the state of the database in the event that it becomes corrupted. For more information about backing up the undercloud database, see Creating a standalone database backup of the undercloud nodes in Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes.
3.4. Updating composable services in custom roles_data
files
This section contains information about new and deprecated composable services.
All nodes
The following services have been deprecated for all nodes. Remove them from all roles.
Service | Reason |
---|---|
| Deprecated services. |
| Deprecated service. |
|
Deprecated in favour of |
| Deprecated service. |
| Deprecated service. |
| Deprecated services. |
| OpenStack Networking (neutron) Load Balancing as a Service deprecated in favour of Octavia. |
| Deprecated services. |
| Deprecated service. |
| This service is removed. |
|
Deprecated in favor of |
| OpenDaylight is no longer supported. |
| Deprecated services. |
| The OpenStack Telemetry services are deprecated in favor of Service Telemetry Framework (STF) for metrics and monitoring. The legacy telemetry services are only available in RHOSP 17.1 to help facilitate the transition to STF and will be removed in a future version of RHOSP. |
| Deprecated service. |
| Deprecated services. |
| Deprecated services. |
| Skydive is no longer supported. |
| Tacker is no longer supported. |
| Deprecated service. |
| Deprecated services. |
| Deprecated service. |
Controller nodes
The following services are new for Controller nodes. Add them to your Controller role.
Service | Reason |
---|---|
| Service for the internal instance of the Image service (glance) API to provide location data to administrators and services that require it, such as the Block Storage service (cinder) and the Compute service (nova). |
Compute nodes
By default, Compute roles run the OS::TripleO::Services::NovaLibvirt
service. Replace this service with the OS::TripleO::Services::NovaLibvirtLegacy
service.
3.5. Network configuration file conversion
If your network configuration templates include the following functions, you must manually convert your NIC templates to Jinja2 Ansible format before you upgrade the overcloud. The following functions are not supported with automatic conversion:
-
'get_file'
-
'get_resource'
-
'digest'
-
'repeat'
-
'resource_facade'
-
'str_replace'
-
'str_replace_strict'
-
'str_split'
-
'map_merge'
-
'map_replace'
-
'yaql'
-
'equals'
-
'if'
-
'not'
-
'and'
-
'or'
-
'filter'
-
'make_url'
-
'contains'
For more information about manually converting your NIC templates, see Manually converting NIC templates to Jinja2 Ansible format in Installing and managing Red Hat OpenStack Platform with director.
3.6. Checking custom Puppet parameters
If you use the ExtraConfig
interfaces for customizations of Puppet parameters, Puppet might report duplicate declaration errors during the upgrade. This is due to changes in the interfaces provided by the puppet modules themselves.
This procedure shows how to check for any custom ExtraConfig
hieradata parameters in your environment files.
Procedure
Select an environment file and the check if it has an
ExtraConfig
parameter:$ grep ExtraConfig ~/templates/custom-config.yaml
-
If the results show an
ExtraConfig
parameter for any role (e.g.ControllerExtraConfig
) in the chosen file, check the full parameter structure in that file. If the parameter contains any puppet Hierdata with a
SECTION/parameter
syntax followed by avalue
, it might have been been replaced with a parameter with an actual Puppet class. For example:parameter_defaults: ExtraConfig: neutron::config::dhcp_agent_config: 'DEFAULT/dnsmasq_local_resolv': value: 'true'
Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:
$ grep dnsmasq_local_resolv
If so, change to the new interface.
The following are examples to demonstrate the change in syntax:
Example 1:
parameter_defaults: ExtraConfig: neutron::config::dhcp_agent_config: 'DEFAULT/dnsmasq_local_resolv': value: 'true'
Changes to:
parameter_defaults: ExtraConfig: neutron::agents::dhcp::dnsmasq_local_resolv: true
Example 2:
parameter_defaults: ExtraConfig: ceilometer::config::ceilometer_config: 'oslo_messaging_rabbit/rabbit_qos_prefetch_count': value: '32'
Changes to:
parameter_defaults: ExtraConfig: oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'
3.7. Final review before upgrade
Complete a final check of all preparation steps before you begin the upgrade.
3.7.1. Upgrade command overview
The upgrade process involves different commands that you run at certain stages of the process.
This section only contains information about each command. You must run these commands in a specific order and provide options specific to your overcloud. Wait until you receive instructions to run these commands at the appropriate step.
3.7.1.1. openstack overcloud upgrade prepare
This command performs the initial preparation steps for the overcloud upgrade, which includes replacing the current overcloud plan on the undercloud with the new OpenStack Platform 17.1 overcloud plan and your updated environment files. This command functions similar to the openstack overcloud deploy
command and uses many of the same options.
Before you run the openstack overcloud upgrade prepare
command, you must perform the overcloud adoption. For more information about overcloud adoption, see Performing the overcloud adoption.
3.7.1.2. openstack overcloud upgrade run
This command performs the upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 17.1 overcloud plan and runs the fast forward tasks on the entire overcloud. This includes running the upgrade process through each OpenStack Platform version from 16.2 to 17.1.
In addition to the standard upgrade process, this command can perform a Leapp upgrade of the operating system on overcloud nodes. Run these tasks using the --tags
option.
Upgrade task tags for Leapp
system_upgrade
-
Task that combines tasks from
system_upgrade_prepare
,system_upgrade_run
, andsystem_upgrade_reboot
. system_upgrade_prepare
- Tasks to prepare for the operating system upgrade with Leapp.
system_upgrade_run
- Tasks to run Leapp and upgrade the operating system.
system_upgrade_reboot
- Tasks to reboot a system and complete the operating system upgrade.
3.7.1.3. openstack overcloud external-upgrade run
This command performs upgrade tasks outside the standard upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 17.1 overcloud plan and you run specific tasks using the --tags
option.
External task tags for container management
container_image_prepare
- Tasks for pulling container images to the undercloud registry and preparing the images for the overcloud to use.
3.7.2. Upgrade Parameters
You can modify the behavior of the upgrade process with upgrade parameters.
Parameter | Description |
---|---|
| Command or script snippet to run on all overcloud nodes to initialize the upgrade process. For example, a repository switch. |
| Common commands required by the upgrades process. This should not normally be modified by the operator and is set and unset in the major-upgrade-composable-steps.yaml and major-upgrade-converge.yaml environment files. |
| Additional command line options to append to the Leapp command. |
|
Print debugging output when running Leapp. The default value is |
| Skip Leapp checks by setting env variables when running Leapp in development/testing. For example, LEAPP_DEVEL_SKIP_RHSM=1. |
|
Use Leapp for operating system upgrade. The default value is |
|
Maximum (seconds) to wait for machine to reboot and respond to a test command. The default value is |
|
Timeout (seconds) for the OS upgrade phase via Leapp. The default value is |
| List of packages to install after Leapp upgrade. |
| List of packages to remove during Leapp upgrade. |
3.7.3. Custom files to include in your deployment
If any overcloud nodes in your deployment are dedicated Object Storage (swift) nodes, you must copy the default roles_data.yaml
file and edit ObjectStorage
to remove deprecated_server_resource_name: 'SwiftStorage'
. Then use the --roles-file
option to pass the file to the openstack overcloud upgrade prepare
command.
3.7.4. New environment files to include with your deployment
In addition to your regular overcloud environment files, you must include new environment files to facilitate the upgrade to Red Hat OpenStack Platform (RHOSP) 17.1.
File | Notes |
---|---|
| This file contains the parameters specific to the upgrade. This file is necessary only for the duration of the upgrade. |
| The file that contains the source and preparation steps. This is the same file that you use with the undercloud upgrade. |
| This file contains the parameters that Ceph Storage is required to override. |
Add these files to the end of your environment file listing when you run the following commands:
-
openstack overcloud upgrade prepare
-
openstack overcloud deploy
3.7.5. Environment files to remove from your deployment
Remove any environment files specific to your OpenStack Platform Red Hat OpenStack Platform 16.2:
- Red Hat OpenStack Platform 16.2 container image list
-
Red Hat OpenStack Platform 16.2 Customer Portal or Satellite
rhel-registration
scripts
Remove these files from the list of environment files you include when you run the following commands:
-
openstack overcloud upgrade prepare
-
openstack overcloud deploy
3.7.6. Upgrading IPA services
If TLS everywhere is enabled in your environment, add an additional permission to the Nova Host Manager role to allow the creation of DNS zone entries.
Prerequisites
Check whether the Nova Host Management permission is included in your environment:
$ ipa privilege-show "Nova Host Management"
If you already have this permission, skip the following procedure.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Add the
Nova Host Management
permission:$ kinit admin $ ipa privilege-add-permission 'Nova Host Management' --permission 'System: Modify Realm Domains'
Create an environment file called
ipa_environment.yaml
and include the following configuration:resource_registry: OS::TripleO::Services::IpaClient: /usr/share/openstack-tripleo-heat-templates/deployment/ipa/ipaservices-baremetal-ansible.yaml parameter_defaults: IdMServer: $IPA_FQDN IdMDomain: $IPA_DOMAIN IdMInstallClientPackages: False
- Save the environment file.
3.7.7. Upgrade checklist
Use the following checklist to determine your readiness to upgrade the overcloud:
Item | Complete |
---|---|
Validated a working overcloud. | Y / N |
Performed a Relax-and-Recover (ReaR) backup of the overcloud control plane. For more information, see Red Hat OpenStack Platform 16.2 Backing up and restoring the undercloud and control plane nodes. | Y / N |
Created a backup of the database that runs on the undercloud node. For more information, see Creating a backup of the undercloud node in Red Hat OpenStack Platform 17.1 Backing up and restoring the undercloud and control plane nodes. | Y / N |
Updated your registration details to Red Hat OpenStack Platform 17.1 repositories and converted your environment file to use the Ansible-based method. | Y / N |
Updated your network configuration templates. | Y / N |
Updated your environment file list with new environment files for Red Hat OpenStack Platform 17.1. | Y / N |
Optional: If your deployment includes dedicated Object Storage (swift) nodes:
Copied the | Y / N |
Removed old environment files only relevant to Red Hat OpenStack Platform 16.2, such as old Red Hat registration and container image location files. | Y / N |
Chapter 4. Performing the overcloud adoption
The overcloud adoption involves the following tasks:
- On each stack, adopt the network and host provisioning configuration exports into the overcloud.
- Define new containers and additional compatibility configuration.
4.1. Running the overcloud upgrade preparation
The upgrade requires running openstack overcloud upgrade prepare
command, which performs the following tasks:
- Updates the overcloud plan to OpenStack Platform 17.1
- Prepares the nodes for the upgrade
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Verify that the following files that were exported during the undercloud upgrade contain the expected configuration for the overcloud upgrade. You can find the following files in the
~/overcloud-deploy
directory:-
tripleo-<stack>-passwords.yaml
-
tripleo-<stack>-network-data.yaml
-
tripleo-<stack>-virtual-ips.yaml
tripleo-<stack>-baremetal-deployment.yaml
NoteIf these files were not generated after the undercloud upgrade, contact Red Hat Support.
-
On the main stack, copy the
passwords.yaml
file to the~/overcloud-deploy/$(<stack>)
directory. Repeat this step on each stack in your environment:$ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-passwords.yaml ~/overcloud-deploy/<stack>/<stack>-passwords.yaml
-
Replace
<stack>
with the name of your stack.
-
Replace
On the main stack, copy the
network-data.yaml
file to the stack user’s home directory and deploy the networks. Repeat this step on each stack in your environment:$ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-network-data.yaml ~/ $ mkdir ~/overcloud_adopt $ openstack overcloud network provision --debug \ --output /home/stack/overcloud_adopt/generated-networks-deployed.yaml tripleo-<stack>-network-data.yaml
For more information, see Provisioning and deploying your overcloud in Installing and managing Red Hat OpenStack Platform with director.
On the main stack, copy the
virtual-ips.yaml
file to the stack user’s home directory and provision the network VIPs. Repeat this step on each stack in your environment:$ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-virtual-ips.yaml ~/ $ openstack overcloud network vip provision --debug \ --stack <stack> --output \ /home/stack/overcloud_adopt/generated-vip-deployed.yaml tripleo-<stack>-virtual-ips.yaml
On the main stack, copy the
baremetal-deployment.yaml
file to the stack user’s home directory and provision the overcloud nodes. Repeat this step on each stack in your environment:$ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-baremetal-deployment.yaml ~/ $ openstack overcloud node provision --debug --stack <stack> \ --output /home/stack/overcloud_adopt/baremetal-deployment.yaml \ tripleo-<stack>-baremetal-deployment.yaml
Complete the following steps to prepare the containers:
Back up the
containers-prepare-parameter.yaml
file that you used for the undercloud upgrade:$ cp containers-prepare-parameter.yaml \ containers-prepare-parameter.yaml.orig
Define the following environment variables before you run the script to update the
containers-prepare-parameter.yaml
file:-
NAMESPACE
: The namespace for the UBI9 images. For example,NAMESPACE='"namespace":"example.redhat.com:5002",'
-
EL8_NAMESPACE
: The namespace for the UBI8 images. -
NEUTRON_DRIVER
: The driver to use and determine which OpenStack Networking (neutron) container to use. Set to the type of containers you used to deploy the original stack. For example, set toNEUTRON_DRIVER='"neutron_driver":"ovn",'
to use OVN-based containers. -
EL8_TAGS
: The tags of the UBI8 images, for example,EL8_TAGS='"tag":"rhel_8_rhosp17.1",'
. -
EL9_TAGS
: The tags of the UBI9 images, for example,EL9_TAGS='"tag":"rhel_9_rhosp17.1",'
. CONTROL_PLANE_ROLES
: The list of control plane roles using the--role
option, for example,--role ControllerOpenstack, --role Database, --role Messaging, --role Networker, --role CephStorage
. To view the list of control plane roles in your environment, run the following command:$ sudo awk '/tripleo_role_name/ {print "--role " $2}' \ /var/lib/mistral/$STACK/tripleo-ansible-inventory.yaml \ | grep -vi compute
COMPUTE_ROLES
: The list of Compute roles using the--role
option, for example,--Compute-1
. To view the list of Compute roles in your environment, run the following command:$ sudo awk '/tripleo_role_name/ {print "--role " $2}' \ /var/lib/mistral/$STACK/tripleo-ansible-inventory.yaml \ | grep -i compute
-
CEPH_TAGS
: If you deployed Red Hat Ceph Storage, specify Red Hat Ceph Storage 5 container images. For example,"ceph_tag":"5-404"
.
-
Run the following script to to update the
containers-prepare-parameter.yaml
file:$ python3 /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \ ${COMPUTE_ROLES} \ ${CONTROL_PLANE_ROLES} \ --enable-multi-rhel \ --excludes collectd \ --excludes nova-libvirt \ --minor-override "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \ --major-override "{${EL9_TAGS}${NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \ --output-env-file \ /home/stack/containers-prepare-parameter.yaml
If you are using Red Hat Ceph Storage, create a file called
ceph_params.yaml
and include the following content:parameter_defaults: CephSpecFqdn: true CephConfigPath: "/etc/ceph" CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms" DeployedCeph: true
NoteIf your Red Hat Ceph Storage deployment includes short names, you must set the
CephSpecFqdn
parameter tofalse
. If set totrue
, the inventory generates with both the short names and domain names, causing the Red Hat Ceph Storage upgrade to fail.Create an environment file called
upgrades-environment.yaml
in your templates directory and include the following content:parameter_defaults: ExtraConfig: nova::workarounds::disable_compute_service_check_for_ffu: true DnsServers: ["<dns_servers>"] DockerInsecureRegistryAddress: <undercloud_FQDN> UpgradeInitCommand: | sudo subscription-manager repos --disable=* if $( grep -q 9.2 /etc/os-release ) then sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms sudo subscription-manager release --set=9.2 else sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms sudo subscription-manager release --set=8.4 fi if $(sudo podman ps | grep -q ceph ) then sudo dnf -y install cephadm fi
-
Replace
<dns_servers>
with a comma-separated list of your DNS server IP addresses, for example,["10.0.0.36", "10.0.0.37"]
. Replace
<undercloud_FQDN>
with the fully qualified domain name (FQDN) of the undercloud host, for example,"undercloud-0.ctlplane.redhat.local:8787"
.For more information about the upgrade parameters that you can configure in the environment file, see Upgrade parameters.
-
Replace
On the undercloud, create a file called
overcloud_upgrade_prepare.sh
in your templates directory. You must create this file for each stack in your environment. This file includes the original content of your overcloud deploy file and the environment files that are relevant to your environment. For example:#!/bin/bash openstack overcloud upgrade prepare --yes \ --timeout 460 \ --templates /usr/share/openstack-tripleo-heat-templates \ --ntp-server 192.168.24.1 \ --stack <stack> \ -r /home/stack/roles_data.yaml \ -e /home/stack/templates/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \ -e /home/stack/templates/network/network-environment.yaml \ -e /home/stack/templates/inject-trust-anchor.yaml \ -e /home/stack/templates/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/templates/debug.yaml \ -e /home/stack/templates/firstboot.yaml \ -e /home/stack/templates/upgrades-environment.yaml \ -e /home/stack/overcloud-params.yaml \ -e /home/stack/overcloud-deploy/overcloud/overcloud-network-environment.yaml \ -e /home/stack/overcloud_adopt/baremetal-deployment.yaml \ -e /home/stack/overcloud_adopt/generated-networks-deployed.yaml \ -e /home/stack/overcloud_adopt/generated-vip-deployed.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \ -e /home/stack/skip_rhel_release.yaml \ -e ~/containers-prepare-parameter.yaml
-
In the original
network-environment.yaml
file (/home/stack/templates/network/network-environment.yaml
), remove all the resource_registry resources that point toOS::TripleO::*::Net::SoftwareConfig
. In the
overcloud_upgrade_prepare.sh
file, include the following options relevant to your environment:-
The environment file (
upgrades-environment.yaml
) with the upgrade-specific parameters (-e
). -
The environment file (
containers-prepare-parameter.yaml
) with your new container image locations (-e
). In most cases, this is the same environment file that the undercloud uses. -
The environment file (
skip_rhel_release.yaml
) with the release parameters (-e). -
Any custom configuration environment files (
-e
) relevant to your deployment. -
If applicable, your custom roles (
roles_data
) file using--roles-file
. -
For Ceph deployments, the environment file (
ceph_params.yaml
) with the Ceph parameters (-e). -
The files that were generated during overcloud adoption (
networks-deployed.yaml
,vip-deployed.yaml
,baremetal-deployment.yaml
) (-e). -
If applicable, the environment file (
ipa-environment.yaml
) with your IPA service (-e). If you use a custom stack name, pass the name with the
--stack
option.NoteYou must include the
nova-hw-machine-type-upgrade.yaml
file in your templates until all of your RHEL 8 Compute nodes are upgraded to RHEL 9 in the environment. If this file is excluded, an error appears in thenova_compute.log
in the/var/log/containers/nova
directory. After you upgrade all of your RHEL 8 Compute nodes to RHEL 9, you can remove this file from your configuration and update the stack.
-
The environment file (
If you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must specify an additional environment file at the end of the
overcloud_upgrade_prepare.sh
script file. You must add the environment file at the end of the script because it overrides another environment file that is specified earlier in the script:-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml
-
In the original
Run the upgrade preparation command for each stack in your environment:
$ source stackrc $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh $ sh /home/stack/overcloud_upgrade_prepare.sh
- Wait until the upgrade preparation completes.
Download the container images:
$ openstack overcloud external-upgrade run --stack <stack> --tags container_image_prepare
Chapter 5. Upgrading an overcloud with director-deployed Ceph deployments
If your environment includes director-deployed Red Hat Ceph Storage deployments, you must upgrade your deployment to Red Hat Ceph Storage 5. With an upgrade to version 5, cephadm
now manages Red Hat Ceph Storage instead of ceph-ansible
.
5.1. Installing ceph-ansible
If you deployed Red Hat Ceph Storage using director, you must complete this procedure. The ceph-ansible
package is required to upgrade Red Hat Ceph Storage with Red Hat OpenStack Platform.
Procedure
Enable the Ceph 5 Tools repository:
[stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
Install the
ceph-ansible
package:[stack@director ~]$ sudo dnf update -y ceph-ansible
5.2. Upgrading to Red Hat Ceph Storage 5
Upgrade the Red Hat Ceph Storage nodes from version 4 to version 5.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Run the Red Hat Ceph Storage external upgrade process with the
ceph
tag:$ openstack overcloud external-upgrade run \ --skip-tags "ceph_health,opendev-validation, ceph_ansible_remote_tmp" \ --stack <stack> \ --tags ceph,facts 2>&1
-
Replace
<stack>
with the name of your stack.
-
Replace
Create the ceph-admin user and distribute the appropriate keyrings:
ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \ ANSIBLE_HOST_KEY_CHECKING=false \ ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \ -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \ -e tripleo_admin_user=ceph-admin \ -e distribute_private_key=true \ --limit ceph_osd,ceph_mon,Undercloud
Update the packages on the Red Hat Ceph Storage nodes:
$ openstack overcloud upgrade run \ --stack <stack> \ --skip-tags ceph_health,opendev-validation,ceph_ansible_remote_tmp \ --tags setup_packages --limit ceph_osd,ceph_mon,Undercloud \ --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1
NoteThe Ceph Monitor service (CephMon) runs on Controller nodes. This command includes the
ceph_mon
tag, which also updates the packages on the Controller nodes.Configure the Red Hat Ceph Storage nodes to use
cephadm
:$ openstack overcloud external-upgrade run \ --skip-tags ceph_health,opendev-validation,ceph_ansible_remote_tmp \ --stack <stack> \ --tags cephadm_adopt 2>&1
Modify the
overcloud_upgrade_prepare.sh
file to replace theceph-ansible
file with acephadm
heat environment file.#!/bin/bash openstack overcloud upgrade prepare --yes \ --timeout 460 \ --templates /usr/share/openstack-tripleo-heat-templates \ --ntp-server 192.168.24.1 \ --stack <stack> \ -r /home/stack/roles_data.yaml \ -e /home/stack/templates/internal.yaml \ … -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \ -e ~/containers-prepare-parameter.yaml
NoteThis example uses the
environments/cephadm/cephadm-rbd-only.yaml
file because RGW is not deployed. If you plan to deploy RGW, useenvironments/cephadm/cephadm.yaml
after you finish upgrading your RHOSP environment, and then run a stack update.Modify the
overcloud_upgrade_prepare.sh
file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml
- Save the file.
Run the upgrade preparation command:
$ source stackrc $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh sh /home/stack/overcloud_upgrade_prepare.sh
The Red Hat Ceph Storage cluster is now upgraded to version 5. This has the following implications:
-
You no longer use
ceph-ansible
to manage Red Hat Ceph Storage. Instead, the Ceph Orchestrator manages the Red Hat Ceph Storage cluster. For more information about the Ceph Orchestrator, see The Ceph Operations Guide. - You no longer need to perform stack updates to make changes to the Red Hat Ceph Storage cluster in most cases. Instead, you can run day two Red Hat Ceph Storage operations directly on the cluster as described in The Ceph Operations Guide. You can also scale Red Hat Ceph Storage cluster nodes up or down as described in Scaling the Ceph Storage cluster in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director.
- Inspect the Red Hat Ceph Storage cluster’s health. For more information about monitoring your cluster’s health, see Monitoring Red Hat Ceph Storage nodes in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director.
Do not include environment files, such as
environments/ceph-ansible/ceph-ansible.yaml
, in openstack deployment commands such asopenstack overcloud deploy
. If your deployment includesceph-ansible
environment files, replace them with one of the following options:Red Hat Ceph Storage deployment Original ceph-ansible
fileCephadm
file replacementCeph RADOS Block Device (RBD) only
Any
ceph-ansible
environment fileenvironments/cephadm/cephadm-rbd-only.yaml
RBD and the Ceph Object Gateway (RGW)
Any
ceph-ansible
environment fileenvironments/cephadm/cephadm.yaml
Ceph Dashboard
environments/ceph-ansible/ceph-dashboard.yaml
Respective file in
environments/cephadm/
Ceph MDS
environments/ceph-ansible/ceph-mds.yaml
Respective file in
environments/cephadm/
-
You no longer use
Chapter 6. Preparing network functions virtualization (NFV)
If you use network functions virtualization (NFV), you must complete some preparation for the overcloud upgrade.
6.1. Network functions virtualization (NFV) environment files
In a typical NFV-based environment, you can enable services such as the following:
- Single-root input/output virtualization (SR-IOV)
- Data Plane Development Kit (DPDK)
You do not require any specific reconfiguration to these services to accommodate the upgrade to Red Hat OpenStack Platform 17.1. However, ensure that the environment files that enable your NFV functionality meet the following requirements:
The default environment files to enable NFV features are located in the
environments/services
directory of the Red Hat OpenStack Platform 17.1openstack-tripleo-heat-templates
collection. If you include the default NFV environment files fromopenstack-tripleo-heat-templates
with your Red Hat OpenStack Platform 16.2 deployment, verify the correct environment file location for the respective feature in Red Hat OpenStack Platform 17.1:-
Open vSwitch (OVS) networking and SR-IOV:
/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml
-
Open vSwitch (OVS) networking and DPDK:
/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml
-
Open vSwitch (OVS) networking and SR-IOV:
-
To maintain OVS compatibility during the upgrade from Red Hat OpenStack Platform 16.2 to Red Hat OpenStack Platform 17.1, you must include the
/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml
environment file. When running deployment and upgrade commands that involve environment files, you must include any NFV-related environment files after theneutron-ovs.yaml
file. For example, when runningopenstack overcloud upgrade prepare
with OVS and NFV environment files, include the files in the following order: - The OVS environment file
- The SR-IOV environment file
The DPDK environment file
$ openstack overcloud upgrade prepare \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs-dpdk.yaml \ ...
There is a migration constraint for NFV workloads: you cannot live migrate instances from OVS-DPDK Compute nodes during an upgrade. Alternatively, you can cold migrate instances from OVS-DPDK Compute nodes during an upgrade.
Chapter 7. Upgrading the overcloud
Upgrade Red Hat OpenStack Platform content across the whole overcloud on each stack in your environment.
7.1. Upgrading RHOSP on all nodes in each stack
Upgrade all overcloud nodes to Red Hat OpenStack Platform (RHOSP) 17.1 for each stack, starting with the main stack.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Upgrade RHOSP on all nodes in your main stack:
$ openstack overcloud upgrade run --yes --stack <stack> --debug --limit allovercloud,undercloud --playbook all
Replace <stack> with the name of the overcloud stack that you want to upgrade the nodes on.
Repeat this step for each stack in your RHOSP deployment.
Chapter 8. Upgrading the undercloud operating system
You must upgrade the undercloud operating system from Red Hat Enterprise Linux 8.4 to Red Hat Enterprise Linux 9.2. The system upgrade performs the following tasks:
- Ensures that network interface naming remains consistent after the system upgrade
- Uses Leapp to upgrade RHEL in-place
- Reboots the undercloud
8.1. Setting the SSH root permission parameter on the undercloud
The Leapp upgrade checks whether the PermitRootLogin
parameter exists in the /etc/ssh/sshd_config
file. You must explicitly set this parameter to either yes
or no
.
For security purposes, set this parameter to no
to disable SSH access to the root user on the undercloud.
Procedure
-
Log in to the undercloud as the
stack
user. Check the
/etc/ssh/sshd_config
file for thePermitRootLogin
parameter:$ sudo grep PermitRootLogin /etc/ssh/sshd_config
If the parameter is not in the
/etc/ssh/sshd_config
file, edit the file and set thePermitRootLogin
parameter:PermitRootLogin no
- Save the file.
8.2. Validating your SSH Key size
Red Hat Enterprise Linux (RHEL) 9.1 requires a minimum SSH key size of 2048 bits. You must check that your current SSH key on Red Hat OpenStack Platform (RHOSP) director has a key of at least 2048 bits. If you do not have an SSH key that meets the requirements of RHEL 9.x, you can lose access to the overcloud.
You must complete this procedure if your SSH key size is under the required 2048 bits.
Procedure
Validate your SSH key size:
ssh-keygen -l -f ~/.ssh/id_rsa.pub
Example output:
1024 SHA256:Xqz0Xz0/aJua6B3qRD7VsLr6n/V3zhmnGSkcFR6FlJw stack@director.example.local (RSA)
If your SSH key is less than 2048 bits, use the
ssh_key_rotation.yaml
ansible playbook to replace the SSH key before continuing.ansible-playbook \ -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml \ /usr/share/ansible/tripleo-playbooks/ssh_key_rotation.yaml \ --extra-vars "keep_old_key_authorized_keys=<true|false> backup_folder_path=</path/to/backup_folder>"
-
Replace
<stack>
with the name of your overcloud stack. -
Replace
<true|false>
based on your backup needs. If you do not include this variable, the default value istrue
. -
Replace
</path/to/backup_folder>
with a backup path. If you do not include this variable, the default value is/home/{{ ansible_user_id }}/backup_keys/{{ ansible_date_time.epoch }}
-
Replace
8.3. Performing the undercloud system upgrade
Upgrade your undercloud operating system to Red Hat Enterprise Linux (RHEL) 9.2. As part of this upgrade, you create a file named system_upgrade.yaml
, which you use to enable the appropriate repositories and required Red Hat OpenStack Platform options and content to install Leapp. You use this file to also upgrade your control plane nodes and Compute nodes.
Procedure
-
Log in to the undercloud as the
stack
user. Create a file named
system_upgrade.yaml
in your templates directory and include the following content:parameter_defaults: UpgradeLeappDevelSkip: "LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_SKIP_CHECK_OS_RELEASE=1 LEAPP_NO_NETWORK_RENAMING=1 LEAPP_DEVEL_TARGET_RELEASE=9.2" UpgradeLeappDebug: false UpgradeLeappEnabled: true LeappActorsToRemove: ['checkifcfg','persistentnetnamesdisable','checkinstalledkernels','biosdevname'] LeappRepoInitCommand: | sudo subscription-manager repos --disable=* subscription-manager repos --enable rhel-8-for-x86_64-baseos-tus-rpms --enable rhel-8-for-x86_64-appstream-tus-rpms --enable openstack-17.1-for-rhel-8-x86_64-rpms subscription-manager release --set=8.4 UpgradeLeappCommandOptions: "--enablerepo=rhel-9-for-x86_64-baseos-eus-rpms --enablerepo=rhel-9-for-x86_64-appstream-eus-rpms --enablerepo=rhel-9-for-x86_64-highavailability-eus-rpms --enablerepo=openstack-17.1-for-rhel-9-x86_64-rpms --enablerepo=fast-datapath-for-rhel-9-x86_64-rpms"
NoteIf your deployment includes Red Hat Ceph Storage nodes, you must add the
CephLeappRepoInitCommand
parameter and specify the source OS version of your Red Hat Ceph Storage nodes. For example:CephLeappRepoInitCommand: ... subscription-manager release --set=8.6
Add the
LeappInitCommand
parameter to yoursystem_upgrade.yaml
file to specify additional requirements applicable to your environment, for example, if you need to define role-based overrides:LeappInitCommand: | sudo subscription-manager repos --disable=* sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms leapp answer --add --section check_vdo.confirm=True dnf -y remove irb
If you use kernel-based NIC names, add the following parameter to the
system_upgrade.yaml
file to ensure that the NIC names persist throughout the upgrade process:parameter_defaults: NICsPrefixesToUdev: ['en'] ...
Run the Leapp upgrade:
$ openstack undercloud upgrade --yes --system-upgrade \ /home/stack/system_upgrade.yaml
NoteIf you need to run the Leapp upgrade again, you must first reset the repositories to RHEL 8.
Reboot the undercloud:
$ sudo reboot
Chapter 9. Upgrading the control plane operating system
Upgrade the operating system on your control plane nodes. The upgrade includes the following tasks:
- Running the overcloud upgrade prepare command with the system upgrade parameters
- Running the overcloud system upgrade, which uses Leapp to upgrade RHEL in-place
- Rebooting the nodes
9.1. Upgrading the control plane nodes
To upgrade the control plane nodes in your environment to Red Hat Enterprise Linux 9.2, you must upgrade one-third of your control plane nodes at a time, starting with the bootstrap nodes.
You upgrade your control plane nodes by using the openstack overcloud upgrade run
command. This command performs the following actions:
- Performs a Leapp upgrade of the operating system.
- Performs a reboot as a part of the Leapp upgrade.
Each node is rebooted during the system upgrade. The performance of the Pacemaker cluster and the Red Hat Ceph Storage cluster is degraded during this downtime, but there is no outage.
This example includes the following nodes with composable roles:
-
controller-0
-
controller-1
-
controller-2
-
database-0
-
database-1
-
database-2
-
networker-0
-
networker-1
-
networker-2
-
ceph-0
-
ceph-1
-
ceph-2
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Run the following script without the
CONTROL_PLANE_ROLES
parameter. Ensure that you include the variables that you used to prepare the containers in Running the overcloud upgrade preparation.python3 \ /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \ ${COMPUTE_ROLES} \ --enable-multi-rhel \ --excludes collectd \ --excludes nova-libvirt \ --minor-override \ "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \ --major-override \ "{${EL9_TAGS}${NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \ --output-env-file \ /home/stack/containers-prepare-parameter.yaml
NoteThe
CONTROL_PLANE_ROLES
parameter defines the list of your control plane roles. Removing this parameter from the script prepares the control plane roles for an upgrade to RHEL 9.2. If theCONTROL_PLANE_ROLES
parameter is included in the script, the control plane roles remain on RHEL 8.4.In the
skip_rhel_release.yaml
file, set theSkipRhelEnforcement
parameter tofalse
:parameter_defaults: SkipRhelEnforcement: false
Update the
overcloud_upgrade_prepare.sh
file:$ openstack overcloud upgrade prepare --yes \ ... -e /home/stack/system_upgrade.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/skip_rhel_release.yaml \ ...
-
Include the
system_upgrade.yaml
file with the upgrade-specific parameters (-e). -
Include the
containers-prepare-parameter.yaml
file with the control plane roles removed (-e). -
Include the
skip_rhel_release.yaml
file with the release parameters (-e).
-
Include the
Run the
overcloud_upgrade_prepare.sh
script:$ sh /home/stack/overcloud_upgrade_prepare.sh
Fetch any new or modified containers that you require for the system upgrade:
$ openstack overcloud external-upgrade run \ --stack <stack> \ --tags container_image_prepare 2>&1
Upgrade the first one-third of the control plane nodes:
$ openstack overcloud upgrade run --yes \ --stack <stack> \ --tags system_upgrade \ --limit <controller-0>,<database-0>,<messaging-0>,<networker-0>,<ceph-0>
-
Replace
<stack>
with the name of your stack. -
Replace
<controller-0>
,<database-0>
,<messaging-0>
,<networker-0>
,<ceph-0>
with your own node names.
-
Replace
Log in to each upgraded node and verify that the cluster in each node is running:
$ sudo pcs status
Repeat this verification step after you upgrade the second one-third of your control plane nodes, and after you upgrade the last one-third of your control plane nodes.
Upgrade the second one-third of the control plane nodes:
$ openstack overcloud upgrade run --yes \ --stack <stack> \ --tags system_upgrade \ --limit <controller-1>,<database-1>,<messaging-1>,<networker-1>,<ceph-1>
-
Replace
<controller-1>
,<database-1>
,<messaging-1>
,<networker-1>
,<ceph-1>
with your own node names.
-
Replace
Upgrade the last one-third of the control plane nodes:
$ openstack overcloud upgrade run --yes \ --stack <stack> \ --tags system_upgrade \ --limit <controller-2>,<database-2>,<messaging-2>,<networker-2>,<ceph-2>
-
Replace
<controller-2>
,<database-2>
,<messaging-2>
,<networker-2>
,<ceph-2>
with your own node names.
-
Replace
Chapter 10. Upgrading the Compute node operating system
You can upgrade the operating system on all of your Compute nodes to RHEL 9.2, or upgrade some Compute nodes while the rest remain on RHEL 8.4.
Prerequisites
10.1. Selecting Compute nodes for upgrade testing
The overcloud upgrade process allows you to either:
- Upgrade all nodes in a role.
- Upgrade individual nodes separately.
To ensure a smooth overcloud upgrade process, it is useful to test the upgrade on a few individual Compute nodes in your environment before upgrading all Compute nodes. This ensures no major issues occur during the upgrade while maintaining minimal downtime to your workloads.
Use the following recommendations to help choose test nodes for the upgrade:
- Select two or three Compute nodes for upgrade testing.
- Select nodes without any critical instances running.
If necessary, migrate critical instances from the selected test Compute nodes to other Compute nodes. Review which migration scenarios are supported:
Source Compute node RHEL version Destination Compute node RHEL version Supported/Not supported RHEL 8
RHEL 8
Supported
RHEL 8
RHEL 9
Supported
RHEL 9
RHEL 9
Supported
RHEL 9
RHEL 8
Not supported
10.2. Upgrading all Compute nodes to RHEL 9.2
Upgrade all your Compute nodes to RHEL 9.2 to take advantage of the latest features and to reduce downtime.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
In the
container-image-prepare.yaml
file, ensure that only the tags specified in theContainerImagePrepare
parameter are included, and theMultiRhelRoleContainerImagePrepare
parameter is removed. For example:parameter_defaults: ContainerImagePrepare: - tag_from_label: "{version}-{release}" set: namespace: name_prefix: name_suffix: tag: rhel_containers: false neutron_driver: ovn ceph_namespace: ceph_image: ceph_tag:
-
In the
roles_data.yaml
file, replace theOS::TripleO::Services::NovaLibvirtLegacy
service with theOS::TripleO::Services::NovaLibvirt
service that is required for RHEL 9.2. Run the
openstack overcloud upgrade prepare
command and include thesystem_upgrade.yaml
file with the upgrade-specific parameters:$ openstack overcloud upgrade prepare --yes … -e /home/stack/system_upgrade.yaml …
Upgrade the operating system on the Compute nodes to RHEL 9.2. Use the
--limit
option with a comma-separated list of nodes that you want to upgrade. The following example upgrades thecompute-0
,compute-1
, andcompute-2
nodes.$ openstack overcloud upgrade run --yes --tags system_upgrade --stack <stack> --limit compute-0,compute-1,compute-2
-
Replace
<stack>
with the name of your stack.
-
Replace
Upgrade the containers on the Compute nodes to RHEL 9.2. Use the
--limit
option with a comma-separated list of nodes that you want to upgrade. The following example upgrades thecompute-0
,compute-1
, andcompute-2
nodes.$ openstack overcloud upgrade run --yes --stack <stack> --limit compute-0,compute-1,compute-2
10.3. Upgrading Compute nodes to a Multi-RHEL environment
You can upgrade a portion of your Compute nodes to RHEL 9.2 while the rest of your Compute nodes remain on RHEL 8.4. This upgrade process involves the following fundamental steps:
-
Plan which nodes you want to upgrade to RHEL 9.2, and which nodes you want to remain on RHEL 8.4. Choose a role name for each role that you are creating for each batch of nodes, for example,
ComputeRHEL-9.2
andComputeRHEL-8.4
. Create roles that store the nodes that you want to upgrade to RHEL 9.2, or the nodes that you want to stay on RHEL 8.4. These roles can remain empty until you are ready to move your Compute nodes to a new role. You can create as many roles as you need and divide nodes among them any way you decide. For example:
-
If your environment uses a role called
ComputeSRIOV
and you need to run a canary test to upgrade to RHEL 9.2, you can create a newComputeSRIOVRHEL9
role and move the canary node to the new role. -
If your environment uses a role called
ComputeOffload
and you want to upgrade most nodes in that role to RHEL 9.2, but keep a few nodes on RHEL 8.4, you can create a newComputeOffloadRHEL8
role to store the RHEL 8.4 nodes. You can then select the nodes in the originalComputeOffload
role to upgrade to RHEL 9.2.
-
If your environment uses a role called
- Move the nodes from each Compute role to the new role.
Upgrade the operating system on specific Compute nodes to RHEL 9.2. You can upgrade nodes in batches from the same role or multiple roles.
NoteIn a Multi-RHEL environment, the deployment should continue to use the pc-i440fx machine type. Do not update the default to Q35. Migrating to the Q35 machine type is a separate, post-upgrade procedure to follow after all Compute nodes are upgraded to RHEL 9.2. For more information about migrating the Q35 machine type, see Updating the default machine type for hosts after an upgrade to RHOSP 17.
Use the following procedures to upgrade Compute nodes to a Multi-RHEL environment:
10.3.1. Creating roles for Multi-RHEL Compute nodes
Create new roles to store the nodes that you are upgrading to RHEL 9.2 or that are staying on RHEL 8.4, and move the nodes into the new roles.
Procedure
Create the relevant roles for your environment. In the
role_data.yaml
file, copy the source Compute role to use for the new role.Repeat this step for each additional role required. Roles can remain empty until you are ready to move your Compute nodes to the new roles.
If you are creating a RHEL 8 role:
name: <ComputeRHEL8> description: | Basic Compute Node role CountDefault: 1 rhsm_enforce_multios: 8.4 ... ServicesDefault: ... - OS::TripleO::Services::NovaLibvirtLegacy
NoteRoles that contain nodes remaining on RHEL 8.4 must include the
NovaLibvirtLegacy
service.-
Replace
<ComputeRHEL8>
with the name of your RHEL 8.4 role. If you are creating a RHEL 9 role:
name: <ComputeRHEL9> description: | Basic Compute Node role CountDefault: 1 ... ServicesDefault: ... - OS::TripleO::Services::NovaLibvirt
NoteRoles that contain nodes being upgraded to RHEL 9.2 must include the
NovaLibvirt
service. ReplaceOS::TripleO::Services::NovaLibvirtLegacy
withOS::TripleO::Services::NovaLibvirt
.- Replace <ComputeRHEL9> with the name of your RHEL 9.2 role.
Copy the
overcloud_upgrade_prepare.sh
file to thecopy_role_Compute_param.sh
file:$ cp overcloud_upgrade_prepare.sh copy_role_Compute_param.sh
Edit the
copy_role_Compute_param.sh
file to include thecopy_role_params.py
script. This script generates the environment file that contains the additional parameters and resources for the new role. For example:/usr/share/openstack-tripleo-heat-templates/tools/copy_role_params.py --rolename-src <Compute_source_role> --rolename-dst <Compute_destination_role> \ -o <Compute_new_role_params.yaml> \ -e /home/stack/templates/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \ -e /home/stack/templates/network/network-environment.yaml \ -e /home/stack/templates/inject-trust-anchor.yaml \ -e /home/stack/templates/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/templates/nodes_data.yaml \ -e /home/stack/templates/debug.yaml \ -e /home/stack/templates/firstboot.yaml \ -e /home/stack/overcloud-params.yaml \ -e /home/stack/overcloud-deploy/overcloud/overcloud-network-environment.yaml \ -e /home/stack/overcloud_adopt/baremetal-deployment.yaml \ -e /home/stack/overcloud_adopt/generated-networks-deployed.yaml \ -e /home/stack/overcloud_adopt/generated-vip-deployed.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \ -e ~/containers-prepare-parameter.yaml
-
Replace
<Compute_source_role>
with the name of your source Compute role that you are copying. -
Replace
<Compute_destination_role>
with the name of your new role. -
Use the -o option to define the name of the output file that includes all the non-default values of the source Compute role for the new role. Replace
<Compute_new_role_params.yaml>
with the name of your output file.
-
Replace
Run the
copy_role_Compute_param.sh
script:$ sh /home/stack/copy_role_Compute_param.sh
Move the Compute nodes from the source role to the new role:
python3 /usr/share/openstack-tripleo-heat-templates/tools/baremetal_transition.py --baremetal-deployment /home/stack/tripleo-<stack>-baremetal-deployment.yaml --src-role <Compute_source_role> --dst-role <Compute_destination_role> <Compute-0> <Compute-1> <Compute-2>
NoteThis tool includes the original
/home/stack/tripleo-<stack>-baremetal-deployment.yaml
file that you exported during the undercloud upgrade. The tool copies and renames the source role definition in the/home/stack/tripleo-<stack>-baremetal-deployment.yaml
file. Then, it changes thehostname_format
to prevent a conflict with the newly created destination role. The tool then moves the node from the source role to the destination role and changes thecount
values.-
Replace
<stack>
with the name of your stack. -
Replace
<Compute_source_role>
with the name of the source Compute role that contains the nodes that you are moving to your new role. -
Replace
<Compute_destination_role>
with the name of your new role. -
Replace
<Compute-0>
<Compute-1>
<Compute-2>
with the names of the nodes that you are moving to your new role.
-
Replace
Reprovision the nodes to update the environment files in the stack with the new role location:
$ openstack overcloud node provision --stack <stack> --output /home/stack/overcloud_adopt/baremetal-deployment.yaml /home/stack/tripleo-<stack>-baremetal-deployment.yaml
NoteThe output
baremetal-deployment.yaml
file is the same file that is used in theovercloud_upgrade_prepare.sh
file during overcloud adoption.Include any Compute roles that are remaining on RHEL 8.4 in the
COMPUTE_ROLES
parameter, and run the following script. For example, if you have a role calledComputeRHEL8
that contains the nodes that are remaining on RHEL 8.4,COMPUTE_ROLES = --role ComputeRHEL8
.python3 /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \ ${COMPUTE_ROLES} \ --enable-multi-rhel \ --excludes collectd --excludes nova-libvirt \ --minor-override "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" --major-override "{${EL9_TAGS}${NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \ --output-env-file /home/stack/containers-prepare-parameter.yaml
- Repeat this procedure to create additional roles and to move additional Compute nodes to those new roles.
10.3.2. Upgrading the Compute node operating system
Upgrade the operating system on selected Compute nodes to RHEL 9.2. You can upgrade multiple nodes from different roles at the same time.
Prerequisites
Ensure that you have created the necessary roles for your environment. For more information about creating roles for a Multi-RHEL environment, see Creating roles for Multi-RHEL Compute nodes.
Procedure
Run the
openstack overcloud upgrade prepare
command to set the parameters and resources required for the new roles.$ openstack overcloud upgrade prepare --yes \ ... -e /home/stack/system_upgrade.yaml \ -e /home/stack/<Compute_new_role_params.yaml> \ ...
-
Include the
system_upgrade.yaml
file with the upgrade-specific parameters (-e). -
Include the environment file that contains the parameters needed for the new role (-e). Replace
<Compute_new_role_params.yaml>
with the name of the environment file you created for your new role. - If you are upgrading nodes from multiple roles at the same time, include the environment file for each new role that you created.
-
Include the
- Optional: Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes and Preparing to migrate.
Upgrade the operating system on specific Compute nodes. Use the
--limit
option with a comma-separated list of nodes that you want to upgrade. The following example upgrades thecomputerhel9-0
,computerhel9-1
,computerhel9-2
, andcomputesriov-42
nodes from theComputeRHEL9
andComputeSRIOV
roles.$ openstack overcloud upgrade run --yes --tags system_upgrade --stack <stack> --limit computerhel9-0,computerhel9-1,computerhel9-2,computesriov-42
- Replace <stack> with the name of your stack.
Upgrade the containers on the Compute nodes to RHEL 9.2. Use the
--limit
option with a comma-separated list of nodes that you want to upgrade. The following example upgrades thecomputerhel9-0
,computerhel9-1
,computerhel9-2
, andcomputesriov-42
nodes from theComputeRHEL9
andComputeSRIOV
roles.$ openstack overcloud upgrade run --yes --stack <stack> --limit computerhel9-0,computerhel9-1,computerhel9-2,computesriov-42
Chapter 11. Performing post-upgrade actions
After you have completed the overcloud upgrade, you must perform some post-upgrade configuration to ensure that your environment is fully supported and ready for future operations.
11.1. Upgrading the overcloud images
You must replace your current overcloud images with new versions. The new images ensure that the director can introspect and provision your nodes using the latest version of Red Hat OpenStack Platform software.
You must use the new version of the overcloud images if you redeploy your overcloud. For more information on installing overcloud images, see Installing the overcloud images in Installing and managing Red Hat OpenStack Platform with director.
Prerequisites
- You have upgraded the undercloud to the latest version.
Procedure
Remove any existing images from the
images
directory on thestack
user’s home (/home/stack/images
):$ rm -rf ~/images/*
Extract the archives:
$ cd ~/images $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-17.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-17.1.tar; do tar -xvf $i; done $ cd ~
11.2. Updating CPU pinning parameters
You must migrate the CPU pinning configuration from the NovaVcpuPinSet
parameter to the following parameters after completing the upgrade to Red Hat OpenStack Platform 17.1:
NovaComputeCpuDedicatedSet
- Sets the dedicated (pinned) CPUs.
NovaComputeCpuSharedSet
- Sets the shared (unpinned) CPUs.
Procedure
-
Log in to the undercloud as the
stack
user. If your Compute nodes support simultaneous multithreading (SMT) but you created instances with the
hw:cpu_thread_policy=isolated
policy, you must perform one of the following options:Unset the
hw:cpu_thread_policy
thread policy and resize the instances:Source your overcloud authentication file:
$ source ~/overcloudrc
Unset the
hw:cpu_thread_policy
property of the flavor:(overcloud) $ openstack flavor unset --property hw:cpu_thread_policy <flavor>
Note-
Unsetting the
hw:cpu_thread_policy
attribute sets the policy to the defaultprefer
policy, which sets the instance to use an SMT-enabled Compute node if available. You can also set thehw:cpu_thread_policy
attribute torequire
, which sets a hard requirements for an SMT-enabled Compute node. -
If the Compute node does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling will fail. To prevent this, set
hw:cpu_thread_policy
toprefer
instead ofrequire
. The defaultprefer
policy ensures that thread siblings are used when available. -
If you use
hw:cpu_thread_policy=isolate
, you must have SMT disabled or use a platform that does not support SMT.
-
Unsetting the
Convert the instances to use the new thread policy.
(overcloud) $ openstack server resize --flavor <flavor> <server> (overcloud) $ openstack server resize confirm <server>
Repeat this step for all pinned instances using the
hw:cpu_thread_policy=isolated
policy.
Migrate instances from the Compute node and disable SMT on the Compute node:
Source your overcloud authentication file:
$ source ~/overcloudrc
Disable the Compute node from accepting new virtual machines:
(overcloud) $ openstack compute service list (overcloud) $ openstack compute service set <hostname> nova-compute --disable
- Migrate all instances from the Compute node. For more information on instance migration, see Migrating virtual machine instances between Compute nodes.
- Reboot the Compute node and disable SMT in the BIOS of the Compute node.
- Boot the Compute node.
Re-enable the Compute node:
(overcloud) $ openstack compute service set <hostname> nova-compute --enable
Source the
stackrc
file:$ source ~/stackrc
-
Edit the environment file that contains the
NovaVcpuPinSet
parameter. Migrate the CPU pinning configuration from the
NovaVcpuPinSet
parameter toNovaComputeCpuDedicatedSet
andNovaComputeCpuSharedSet
:-
Migrate the value of
NovaVcpuPinSet
toNovaComputeCpuDedicatedSet
for hosts that were previously used for pinned instances. -
Migrate the value of
NovaVcpuPinSet
toNovaComputeCpuSharedSet
for hosts that were previously used for unpinned instances. -
If there is no value set for NovaVcpuPinSet, then all Compute node cores should be assigned to either
NovaComputeCpuDedicatedSet
orNovaComputeCpuSharedSet
, depending on the type of instances you intend to host on the nodes.
For example, your previous environment file might contain the following pinning configuration:
parameter_defaults: ... NovaVcpuPinSet: 1,2,3,5,6,7 ...
To migrate the configuration to a pinned configuration, set the
NovaComputeCpuDedicatedSet
parameter and unset theNovaVcpuPinSet
parameter:parameter_defaults: ... NovaComputeCpuDedicatedSet: 1,2,3,5,6,7 NovaVcpuPinSet: "" ...
To migrate the configuration to an unpinned configuration, set the
NovaComputeCpuSharedSet
parameter and unset theNovaVcpuPinSet
parameter:parameter_defaults: ... NovaComputeCpuSharedSet: 1,2,3,5,6,7 NovaVcpuPinSet: "" ...
ImportantEnsure the configuration of either
NovaComputeCpuDedicatedSet
orNovaComputeCpuSharedSet
matches the configuration defined inNovaVcpuPinSet
. To change the configuration for either of these, or to configure bothNovaComputeCpuDedicatedSet
orNovaComputeCpuSharedSet
, ensure the Compute nodes with the pinning configuration are not running any instances before updating the configuration.-
Migrate the value of
- Save the file.
Run the deployment command to update the overcloud with the new CPU pinning parameters.
(undercloud) $ openstack overcloud deploy \ --stack _STACK NAME_ \ --templates \ ... -e /home/stack/templates/<compute_environment_file>.yaml ...
Additional resources
11.3. Updating the default machine type for hosts after an upgrade to RHOSP 17
The machine type of an instance is a virtual chipset that provides certain default devices, such as a PCIe graphics card or Ethernet controller. Cloud users can specify the machine type for their instances by using an image with the hw_machine_type
metadata property that they require.
Cloud administrators can use the Compute parameter NovaHWMachineType
to configure each Compute node architecture with a default machine type to apply to instances hosted on that architecture. If the hw_machine_type
image property is not provided when launching the instance, the default machine type for the host architecture is applied to the instance. Red Hat OpenStack Platform (RHOSP) 17 is based on RHEL 9. The pc-i440fx
QEMU machine type is deprecated in RHEL 9, therefore the default machine type for x86_64
instances that run on RHEL 9 has changed from pc
to q35
. Based on this change in RHEL 9, the default value for machine type x86_64
has also changed from pc
in RHOSP 16 to q35
in RHOSP 17.
From RHOSP 16.2 and later, the Compute service records the instance machine type within the system metadata of the instance when it launches an instance. This means that it is now possible to change the NovaHWMachineType
during the lifetime of a RHOSP deployment without affecting the machine type of existing instances.
The Compute service records the machine type of instances that are not in a SHELVED_OFFLOADED
state. Therefore, after an upgrade to RHOSP 17 you must manually record the machine type of instances that are in SHELVED_OFFLOADED
state, and verify that all instances within the environment or specific cell have had a machine type recorded. After you have updated the system metadata for each instance with the machine types, you can update the NovaHWMachineType
parameter to the RHOSP 17 default, q35
, without affecting the machine type of existing instances.
Prerequisites
- Upgrade all Compute nodes to RHEL 9.2. For more information about upgrading Compute nodes, see Upgrading all Compute nodes to RHEL 9.2.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file.$ source ~/stackrc
Log in to a Controller node as the
heat-admin
user:(undercloud)$ metalsmith list $ ssh heat-admin@<controller_ip>
Replace
<controller_ip>
with the IP address of the Controller node.Retrieve the list of instances that have no machine type set:
[heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \ nova-manage libvirt list_unset_machine_type
Check the
NovaHWMachineType
parameter in thenova-hw-machine-type-upgrade.yaml
file for the default machine type for the instance host. The default value for theNovaHWMachineType
parameter in RHOSP 16.2 is as follows:x86_64=pc-i440fx-rhel7.6.0,aarch64=virt-rhel7.6.0,ppc64=pseries-rhel7.6.0,ppc64le=pseries-rhel7.6.0
Update the system metadata of each instance with the default instance machine type:
[heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \ nova-manage libvirt update_machine_type <instance_uuid> <machine_type>
-
Replace
<instance_uuid>
with the UUID of the instance. -
Replace
<machine_type>
with the machine type to record for the instance.
-
Replace
Confirm that the machine type is recorded for all instances:
[heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \ nova-status upgrade check
This command returns a warning if an instance is found without a machine type. If you get this warning, repeat this procedure from step 4.
-
Change the default value of
NovaHWMachineType
in a Compute environment file tox86_64=q35
and deploy the overcloud.
Verification
Create an instance that has the default machine type:
(overcloud)$ openstack server create --flavor <flavor> \ --image <image> --network <network> \ --wait defaultMachineTypeInstance
-
Replace
<flavor>
with the name or ID of a flavor for the instance. -
Replace
<image>
with the name or ID of an image that does not sethw_machine_type
. -
Replace
<network>
with the name or ID of the network to connect the instance to.
-
Replace
Verify that the instance machine type is set to the default value:
[heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \ nova-manage libvirt get_machine_type <instance_uuid>
Replace
<instance_uuid>
with the UUID of the instance.Hard reboot an instance with a machine type of
x86_64=pc-i440fx
:(overcloud)$ openstack server reboot --hard <instance_uuid>
Replace
<instance_uuid>
with the UUID of the instance.Verify that the instance machine type has not been changed:
[heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \ nova-manage libvirt get_machine_type <instance_uuid>
Replace
<instance_uuid>
with the UUID of the instance.
11.4. Re-enabling fencing in the overcloud
Before you upgraded the overcloud, you disabled fencing in Disabling fencing in the overcloud. After you upgrade your environment, re-enable fencing to protect your data if a node fails.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Log in to a Controller node and run the Pacemaker command to re-enable fencing:
$ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true"
-
Replace
<controller_ip>
with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with theopenstack server list
command.
-
Replace
-
In the
fencing.yaml
environment file, set theEnableFencing
parameter totrue
.
Additional Resources