Chapter 1. Preparing for a minor update
Keep your Red Hat OpenStack Platform (RHOSP) 16.2 environment updated with the latest packages and containers.
You can update the following versions:
Old RHOSP Version | New RHOSP Version |
---|---|
Red Hat OpenStack Platform 16.1.z | Red Hat OpenStack Platform 16.2 latest |
Red Hat OpenStack Platform 16.2.z | Red Hat OpenStack Platform 16.2 latest |
The RHOSP minor update process workflow
You must complete the following steps to update your RHOSP environment:
- Prepare your environment for the RHOSP minor update.
- Update the undercloud to the latest OpenStack 16.2.z version.
- Update the overcloud to the latest OpenStack 16.2.z version.
- Upgrade all Red Hat Ceph Storage services.
- Run the convergence command to refresh your overcloud stack.
If you have a multistack infrastructure, update each overcloud stack completely, one at a time. If you have a distributed compute node (DCN) infrastructure, update the overcloud at the central location completely, and then update the overcloud at each edge site, one at a time.
Considerations before you update your RHOSP environment
To help guide you during the update process, consider the following information:
- Red Hat recommends backing up the undercloud and overcloud control planes. For more information about backing up nodes, see Backing up and restoring the undercloud and control plane nodes.
- Familiarize yourself with the known issues that might block an update.
- Familiarize yourself with the possible update and upgrade paths before you begin your update. For more information, see Section 1.1, “Upgrade paths for long life releases”.
-
To identify your current maintenance release, run
$ cat /etc/rhosp-release
. You can also run this command after updating your environment to validate the update.
Known issues that might block an update
Review the following known issues that might affect a successful minor version update.
During some updates from 16.1 to 16.2.6, the collectd
container (sensubility) uses more memory than required, which causes a podman-initiated restart. If a podman-initiated restart occurs during an update, the update fails.
If a podman-initiated restart of the collectd
container occurs during an update, you must disable the collectd
container, and then enable the collectd
container after a successful update. For more information about disabling and enabling the collectd
container, see the following Red Hat Knowledgebase solution Updates fail because collectd container (sensubility) runs OOM.
Overcloud nodes that run Pacemaker version 2.0.3-5.el8_2.4
might fail to update successfully because of a race condition that occurs when shutting down the cluster on a node.
If Pacemaker version 2.0.3-5.el8_2.4
is currently installed on any of the overcloud nodes, you must upgrade Pacemaker before you can update the overcloud nodes. For more information, see the following Red Hat Knowledgebase solution Update from OSP16.1 to OSP16.2 might fail to update certain HA containers.
Starting with Red Hat Enterprise Linux (RHEL) version 8.3, support for the Intel Transactional Synchronization Extensions (TSX) feature is disabled by default. This causes issues with instance live migration between hosts in the following migration scenario:
- Migrating from hosts where the TSX kernel argument is enabled to hosts where the TSX kernel argument is disabled.
Live migration can be unsuccessful in Intel hosts that support the TSX feature. For more information about the CPUs that are affected by this issue, see Affected Configurations.
For more information, review the following Red Hat Knowledgebase solution Guidance on Intel TSX impact on OpenStack guests.
For nodes that run RHEL 8.4, and are based on composable roles, you must update the Database
role first before you can update any other role.
There is a known issue with the advanced-virt-for-rhel-8-x86_64-eus-rpms
and advanced-virt-for-rhel-8-x86_64-rpms
repositories that prevents a successful upgrade. To disable these repositories before upgrading, see the Red Hat Knowledgebase solution advanced-virt-for-rhel-8-x86_64-rpms are no longer required in OSP 16.2.
There is a known issue with upgrading from RHOSP 16.1 to 16.2, and with upgrading from RHOSP 16.2.1 to 16.2.2, related to changes in Podman and the libvirt service. If you do not migrate workloads before upgrading, then the upgrade might fail.
Do not update from RHOSP 16.2.0 to 16.2.2 or 16.2.3 until you evaluate your risk of serious impact from a libvirt version incompatibility.
To evaluate your risk, complete the following steps:
Check the libvirt package in the
nova_libvirt
container on all Compute nodes:$ sudo podman exec nova_libvirt rpm -qa libvirt-*
Check the libvirt version of the
nova_compute
container:$ sudo podman exec nova_compute rpm -qa libvirt-*
If the libvirt version is 7.0, the deployment is not affected by the bug. You can perform the update.
If the libvirt version is 7.6, the deployment is affected by the bug. Your update is at risk. To update your deployment, follow the steps in Workaround for a libvirt version-compat issue (bug 2109350) when updating RHOSP 16.2.0.
In Red Hat OpenStack Platform (RHOSP) 16.2, the nova::dhcp_domain
parameter was introduced. If you update from RHOSP 16.1 to any 16.2 release, and your custom template includes the legacy nova::metadata::dhcp_domain
parameter, a conflict occurs with the nova::dhcp_domain
parameter. As a result, hostnames do not generate on Compute nodes. To avoid this issue, choose one of the following options:
-
Set the legacy
nova::metadata::dhcp_domain
andnova::dhcp_domain
parameters to the same value. - Wait to update. A fix is planned in RHOSP 16.2.6.
Do not alter OVN DB entries during an update from RHOSP 16.1 to 16.2, if the update includes an OVN DB schema upgrade. Doing so can result in misconfiguration and data loss.
If you alter the OVN DB during an update that includes an OVN DB schema upgrade and OpenShift, Kuryr, and the Load-balancing service (octavia), you might not be able to delete Load-balancing entities.
Workaround: If you altered the OVN DB during an update that includes an OVN DB schema upgrade and OpenShift, Kuryr, and the Load-balancing service, and you cannot delete Load-balancing entities, perform the following steps:
- Access the mysql octavia DB.
-
Change the entity’s
provisioning_status
toDELETED
.
If the issue occurs over any other OVN DB entity after altering the OVN DB during an update, run the neutron-db-sync tool
.
Procedure
To prepare your RHOSP environment for the minor update, complete the following procedures:
- Section 1.2, “Locking the environment to a Red Hat Enterprise Linux release”
- Section 1.3, “Switching to TUS repositories”
- Section 1.4, “Updating Red Hat Openstack Platform and Ansible repositories”
- Section 1.5, “Setting the container-tools module version”
- Section 1.6, “Updating the container image preparation file”
- Section 1.7, “Updating the SSL/TLS configuration”
- Section 1.8, “Disabling fencing in the overcloud”
1.1. Upgrade paths for long life releases
Familiarize yourself with the possible update and upgrade paths before you begin an update or an upgrade.
You can view your current RHOSP and RHEL versions in the /etc/rhosp-release
and /etc/redhat-release
files.
Table 1.1. Updates version path
Current version | Target version |
---|---|
RHOSP 10.0.x on RHEL 7.x | RHOSP 10.0 latest on RHEL 7.7 latest |
RHOSP 13.0.x on RHEL 7.x | RHOSP 13.0 latest on RHEL 7.9 latest |
RHOSP 16.1.x on RHEL 8.2 | RHOSP 16.1 latest on RHEL 8.2 latest |
RHOSP 16.1.x on RHEL 8.2 | RHOSP 16.2 latest on RHEL 8.4 latest |
RHOSP 16.2.x on RHEL 8.4 | RHOSP 16.2 latest on RHEL 8.4 latest |
Table 1.2. Upgrades version path
Current version | Target version |
---|---|
RHOSP 10 on RHEL 7.7 | RHOSP 13 latest on RHEL 7.9 latest |
RHOSP 13 on RHEL 7.9 | RHOSP 16.1 latest on RHEL 8.2 latest |
RHOSP 13 on RHEL 7.9 | RHOSP 16.2 latest on RHEL 8.4 latest |
For more information, see Framework for Upgrades (13 to 16.2).
Red Hat provides two options for upgrading your environment to the next long life release:
- In-place upgrade
- Perform an upgrade of the services in your existing environment. This guide primarily focuses on this option.
- Parallel migration
- Create a new Red Hat OpenStack Platform 16.2 environment and migrate your workloads from your current environment to the new environment. For more information about Red Hat OpenStack Platform parallel migration, contact Red Hat Global Professional Services.
The durations in this table are minimal estimates based on internal testing and might not apply to all productions environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. To accurately gauge the upgrade duration for each task, perform these procedures in a test environment with hardware similar to your production environment.
Table 1.3. Impact and duration of upgrade paths
In-place upgrade | Parallel migration | |
---|---|---|
Upgrade duration for undercloud | Estimated duration for each major action includes the following:
| None. You are creating a new undercloud in addition to your existing undercloud. |
Upgrade duration for overcloud control plane | Estimates for each Controller node:
| None. You are creating a new control plane in addition to your existing control plane. |
Outage duration for control plane | The duration of the service upgrade of the bootstrap Controller node, which is approximately 60 minutes. | None. Both overclouds are operational during the workload migration. |
Consequences of control plane outage | You cannot perform OpenStack operations during the outage. | No outage. |
Upgrade duration for overcloud data plane | Estimates for each Compute node and Ceph Storage node:
| None. You are creating a new data plane in addition to your existing data plane. |
Outage duration for data plane | The outage is minimal due to workload migration from node to node. | The outage is minimal due to workload migration from overcloud to overcloud. |
Additional hardware requirements | No additional hardware is required. | Additional hardware is required to create a new undercloud and overcloud. |
1.2. Locking the environment to a Red Hat Enterprise Linux release
Red Hat OpenStack Platform (RHOSP) 16.2 is supported on Red Hat Enterprise Linux (RHEL) 8.4. Before you perform the update, lock the undercloud and overcloud repositories to the RHEL 8.4 release to avoid upgrading the operating system to a newer minor release.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
-
Edit your overcloud subscription management environment file, which is the file that contains the
RhsmVars
parameter. The default name for this file is usuallyrhsm.yml
. Check if your subscription management configuration includes the
rhsm_release
parameter. If therhsm_release
parameter is not present, add it and set it to 8.4:parameter_defaults: RhsmVars: … rhsm_username: "myusername" rhsm_password: "p@55w0rd!" rhsm_org_id: "1234567" rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd" rhsm_method: "portal" rhsm_release: "8.4"
- Save the overcloud subscription management environment file.
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml
If you use an overcloud name that is different to the default overcloud name of
overcloud
, set the name of your overcloud with the--plan
option.Create a playbook that contains a task to lock the operating system version to RHEL 8.4 on all nodes:
$ cat > ~/set_release.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: set release to 8.4 command: subscription-manager release --set=8.4 become: true EOF
Run the
set_release.yaml
playbook:$ ansible-playbook -i ~/inventory.yaml -f 25 ~/set_release.yaml --limit <undercloud>,<Controller>,<Compute>
-
Use the
--limit
option to apply the content to all RHOSP nodes. Replace<undercloud>
,<Controller>
,<Compute>
with the Ansible groups in your environment that contain those nodes. - You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.
-
Use the
To manually lock a node to a version, log in to the node and run the subscription-manager release
command:
$ sudo subscription-manager release --set=8.4
1.3. Switching to TUS repositories
Your Red Hat OpenStack Platform (RHOSP) subscription includes repositories for Red Hat Enterprise Linux (RHEL) 8.4 Extended Update Support (EUS), in addition to standard repositories. After May 30, 2023, you must enable the RHEL 8.4 Telecommunications Update Service (TUS) repositories for Maintenance Support. The TUS repositories include the latest security patches and bug fixes for RHEL 8.4.
Switch your repositories to the required TUS repositories before you perform an update.
Table 1.4. Switching from EUS repositories to TUS repositories
EUS repository | TUS repository |
---|---|
rhel-8-for-x86_64-baseos-eus-rpms | rhel-8-for-x86_64-baseos-tus-rpms |
rhel-8-for-x86_64-appstream-eus-rpms | rhel-8-for-x86_64-appstream-tus-rpms |
rhel-8-for-x86_64-highavailability-eus-rpms | rhel-8-for-x86_64-highavailability-tus-rpms |
Table 1.5. Switching from standard repositories to TUS repositories
Standard repository | TUS repository |
---|---|
rhel-8-for-x86_64-baseos-rpms | rhel-8-for-x86_64-baseos-tus-rpms |
rhel-8-for-x86_64-appstream-rpms | rhel-8-for-x86_64-appstream-tus-rpms |
rhel-8-for-x86_64-highavailability-rpms | rhel-8-for-x86_64-highavailability-tus-rpms |
You must use TUS repositories to retain compatibility with a specific version of Podman. Later versions of Podman are untested with Red Hat OpenStack Platform 16.2 and can cause unexpected results.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
-
Edit your overcloud subscription management environment file, which is the file that contains the
RhsmVars
parameter. The default name for this file is usuallyrhsm.yml
. Check the
rhsm_repos
parameter in your subscription management configuration. If this parameter does not include the TUS repositories, change the relevant repositories to the TUS versions:parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-tus-rpms - rhel-8-for-x86_64-appstream-tus-rpms - rhel-8-for-x86_64-highavailability-tus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - rhceph-4-tools-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms
- Save the overcloud subscription management environment file.
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml
If you use an overcloud name that is different to the default overcloud name of
overcloud
, set the name of your overcloud with the--plan
option.Create a playbook that contains a task to set the repositories to RHEL 8.4 TUS on all nodes:
$ cat > ~/change_tus.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change to tus repos command: subscription-manager repos --disable=rhel-8-for-x86_64-baseos-eus-rpms --disable=rhel-8-for-x86_64-appstream-eus-rpms --disable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms become: true EOF
If your environment includes standard repositories, disable the following repositories:
- rhel-8-for-x86_64-baseos-rpms
- rhel-8-for-x86_64-appstream-rpms
- rhel-8-for-x86_64-highavailability-rpms
Run the
change_tus.yaml
playbook:$ ansible-playbook -i ~/inventory.yaml -f 25 ~/change_tus.yaml --limit <undercloud>,<Controller>,<Compute>
-
Use the
--limit
option to apply the content to all Red Hat OpenStack Platform nodes. Replace<undercloud>
,<Controller>
,<Compute>
with the Ansible groups in your environment that contain those nodes. - You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.
-
Use the
1.4. Updating Red Hat Openstack Platform and Ansible repositories
Update your repositories to use Red Hat OpenStack Platform (RHOSP) 16.2 and Ansible 2.9 packages.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
-
Edit your overcloud subscription management environment file, which is the file that contains the
RhsmVars
parameter. The default name for this file is usuallyrhsm.yml
. Check the
rhsm_repos
parameter in your subscription management configuration. If therhsm_repos
parameter uses the RHOSP 16.1 and Ansible 2.8 repositories, change the repository to the correct versions:parameter_defaults: RhsmVars: rhsm_repos: - rhel-8-for-x86_64-baseos-tus-rpms - rhel-8-for-x86_64-appstream-tus-rpms - rhel-8-for-x86_64-highavailability-tus-rpms - ansible-2.9-for-rhel-8-x86_64-rpms - openstack-16.2-for-rhel-8-x86_64-rpms - fast-datapath-for-rhel-8-x86_64-rpms
- Save the overcloud subscription management environment file.
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml
If you use an overcloud name that is different to the default overcloud name of
overcloud
, set the name of your overcloud with the--plan
option.Create a playbook that contains a task to set the repositories to RHOSP 16.2 on all RHOSP nodes:
$ cat > ~/update_rhosp_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change osp repos command: subscription-manager repos --disable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF
Run the
update_rhosp_repos.yaml
playbook:$ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_rhosp_repos.yaml --limit <undercloud>,<Controller>,<Compute>
-
Use the
--limit
option to apply the content to all RHOSP nodes. Replace<undercloud>
,<Controller>
,<Compute>
with the Ansible groups in your environment that contain those nodes. - You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.
-
Use the
Create a playbook that contains a task to set the repositories to RHOSP 16.2 on all Ceph Storage nodes:
$ cat > ~/update_ceph_repos.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: change ceph repos command: subscription-manager repos --disable=openstack-16-deployment-tools-for-rhel-8-x86_64-rpms --enable=openstack-16.2-deployment-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms become: true EOF
Run the
update_ceph_repos.yaml
playbook:$ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_ceph_repos.yaml --limit CephStorage
Use the
--limit
option to apply the content to Ceph Storage nodes.
1.5. Setting the container-tools module version
Set the container-tools
module to version 3.0
to ensure that you use the correct package versions on all nodes.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
Create a static inventory file of your overcloud:
$ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml
If you use an overcloud name that is different to the default overcloud name of
overcloud
, set the name of your overcloud with the--plan
option.Create a playbook that contains a task to set the
container-tools
module to version3.0
on all nodes:$ cat > ~/container-tools.yaml <<'EOF' - hosts: all gather_facts: false tasks: - name: disable default dnf module for container-tools command: dnf module reset -y container-tools become: true - name: set dnf module for container-tools:3.0 command: dnf module enable -y container-tools:3.0 become: true EOF
Run the
container-tools.yaml
playbook against all nodes:$ ansible-playbook -i ~/inventory.yaml -f 25 ~/container-tools.yaml
1.6. Updating the container image preparation file
The container preparation file is the file that contains the ContainerImagePrepare
parameter. You use this file to define the rules for obtaining container images for the undercloud and overcloud.
Before you update your environment, check the file to ensure that you obtain the correct image versions.
Procedure
-
Edit the container preparation file. The default name for this file is usually
containers-prepare-parameter.yaml
. Check the
tag
parameter is set to16.2
for each rule set:parameter_defaults: ContainerImagePrepare: - push_destination: true set: … tag: '16.2' tag_from_label: '{version}-{release}'
NoteIf you do not want to use a specific tag for the update, such as
16.2
or16.2.2
, remove thetag
key-value pair and specifytag_from_label
only. This uses the installed Red Hat OpenStack Platform version to determine the value for the tag to use as part of the update process.- Save this file.
1.7. Updating the SSL/TLS configuration
Remove the NodeTLSData
resource from the resource_registry
to update your SSL/TLS configuration.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
-
Edit your custom overcloud SSL/TLS public endpoint file, which is usually named
~/templates/enable-tls.yaml
. Remove the
NodeTLSData
resource from theresource_registry
:resource_registry: OS::TripleO::NodeTLSData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/tls-cert-inject.yaml ...
The overcloud deployment uses a new service in HAProxy to determine if SSL/TLS is enabled.
NoteIf this is the only resource in the
resource_registry
section of theenable-tls.yaml
file, remove the completeresource_registry
section.- Save the SSL/TLS public endpoint file.
If you are updating from Red Hat OpenStack Platform 16.1, you must update the permissions in Red Hat Identity Manager (IdM) for all pre-update checks to pass. Use
ssh
to login to the server that is running your IdM, and then run the following commands:$ kinit admin $ ipa privilege-add-permission 'Nova Host Management' --permission 'System: Modify Realm Domains'
1.8. Disabling fencing in the overcloud
Before you update the overcloud, ensure that fencing is disabled.
If fencing is deployed in your environment during the Controller nodes update process, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results.
If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the update to avoid any unintended results.
Procedure
-
Log in to the undercloud as the
stack
user. Source the
stackrc
file.$ source ~/stackrc
Log in to a Controller node and run the Pacemaker command to disable fencing:
$ ssh heat-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"
Replace
<controller_ip>
with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with theopenstack server list
command.-
In the
fencing.yaml
environment file, set theEnableFencing
parameter tofalse
to ensure that fencing stays disabled during the update process.
Additional Resources