Chapter 1. Preparing for a minor update

Keep your Red Hat OpenStack Platform (RHOSP) 16.2 environment updated with the latest packages and containers.

You can update the following versions:

Old RHOSP VersionNew RHOSP Version

Red Hat OpenStack Platform 16.1.z

Red Hat OpenStack Platform 16.2 latest

Red Hat OpenStack Platform 16.2.z

Red Hat OpenStack Platform 16.2 latest

The RHOSP minor update process workflow

You must complete the following steps to update your RHOSP environment:

  1. Prepare your environment for the RHOSP minor update.
  2. Update the undercloud to the latest OpenStack 16.2.z version.
  3. Update the overcloud to the latest OpenStack 16.2.z version.
  4. Upgrade all Red Hat Ceph Storage services.
  5. Run the convergence command to refresh your overcloud stack.

If you have a multistack infrastructure, update each overcloud stack completely, one at a time. If you have a distributed compute node (DCN) infrastructure, update the overcloud at the central location completely, and then update the overcloud at each edge site, one at a time.

Considerations before you update your RHOSP environment

To help guide you during the update process, consider the following information:

  • Red Hat recommends backing up the undercloud and overcloud control planes. For more information about backing up nodes, see Backing up and restoring the undercloud and control plane nodes.
  • Familiarize yourself with the known issues that might block an update.
  • Familiarize yourself with the possible update and upgrade paths before you begin your update. For more information, see Section 1.1, “Upgrade paths for long life releases”.
  • To identify your current maintenance release, run $ cat /etc/rhosp-release. You can also run this command after updating your environment to validate the update.

Known issues that might block an update

Review the following known issues that might affect a successful minor version update.

During some updates from 16.1 to 16.2.6, the collectd container (sensubility) uses more memory than required, which causes a podman-initiated restart. If a podman-initiated restart occurs during an update, the update fails.

If a podman-initiated restart of the collectd container occurs during an update, you must disable the collectd container, and then enable the collectd container after a successful update. For more information about disabling and enabling the collectd container, see the following Red Hat Knowledgebase solution Updates fail because collectd container (sensubility) runs OOM.

Overcloud nodes that run Pacemaker version 2.0.3-5.el8_2.4 might fail to update successfully because of a race condition that occurs when shutting down the cluster on a node.

If Pacemaker version 2.0.3-5.el8_2.4 is currently installed on any of the overcloud nodes, you must upgrade Pacemaker before you can update the overcloud nodes. For more information, see the following Red Hat Knowledgebase solution Update from OSP16.1 to OSP16.2 might fail to update certain HA containers.

Starting with Red Hat Enterprise Linux (RHEL) version 8.3, support for the Intel Transactional Synchronization Extensions (TSX) feature is disabled by default. This causes issues with instance live migration between hosts in the following migration scenario:

  • Migrating from hosts where the TSX kernel argument is enabled to hosts where the TSX kernel argument is disabled.

Live migration can be unsuccessful in Intel hosts that support the TSX feature. For more information about the CPUs that are affected by this issue, see Affected Configurations.

For more information, review the following Red Hat Knowledgebase solution Guidance on Intel TSX impact on OpenStack guests.

For nodes that run RHEL 8.4, and are based on composable roles, you must update the Database role first before you can update any other role.

There is a known issue with the advanced-virt-for-rhel-8-x86_64-eus-rpms and advanced-virt-for-rhel-8-x86_64-rpms repositories that prevents a successful upgrade. To disable these repositories before upgrading, see the Red Hat Knowledgebase solution advanced-virt-for-rhel-8-x86_64-rpms are no longer required in OSP 16.2.

There is a known issue with upgrading from RHOSP 16.1 to 16.2, and with upgrading from RHOSP 16.2.1 to 16.2.2, related to changes in Podman and the libvirt service. If you do not migrate workloads before upgrading, then the upgrade might fail.

Do not update from RHOSP 16.2.0 to 16.2.2 or 16.2.3 until you evaluate your risk of serious impact from a libvirt version incompatibility.

To evaluate your risk, complete the following steps:

  1. Check the libvirt package in the nova_libvirt container on all Compute nodes:

    $ sudo podman exec nova_libvirt rpm -qa libvirt-*
  2. Check the libvirt version of the nova_compute container:

    $ sudo podman exec nova_compute rpm -qa libvirt-*

If the libvirt version is 7.0, the deployment is not affected by the bug. You can perform the update.

If the libvirt version is 7.6, the deployment is affected by the bug. Your update is at risk. To update your deployment, follow the steps in Workaround for a libvirt version-compat issue (bug 2109350) when updating RHOSP 16.2.0.

In Red Hat OpenStack Platform (RHOSP) 16.2, the nova::dhcp_domain parameter was introduced. If you update from RHOSP 16.1 to any 16.2 release, and your custom template includes the legacy nova::metadata::dhcp_domain parameter, a conflict occurs with the nova::dhcp_domain parameter. As a result, hostnames do not generate on Compute nodes. To avoid this issue, choose one of the following options:

  • Set the legacy nova::metadata::dhcp_domain and nova::dhcp_domain parameters to the same value.
  • Wait to update. A fix is planned in RHOSP 16.2.6.

Do not alter OVN DB entries during an update from RHOSP 16.1 to 16.2, if the update includes an OVN DB schema upgrade. Doing so can result in misconfiguration and data loss.

If you alter the OVN DB during an update that includes an OVN DB schema upgrade and OpenShift, Kuryr, and the Load-balancing service (octavia), you might not be able to delete Load-balancing entities.

Workaround: If you altered the OVN DB during an update that includes an OVN DB schema upgrade and OpenShift, Kuryr, and the Load-balancing service, and you cannot delete Load-balancing entities, perform the following steps:

  1. Access the mysql octavia DB.
  2. Change the entity’s provisioning_status to DELETED.

If the issue occurs over any other OVN DB entity after altering the OVN DB during an update, run the neutron-db-sync tool.

Procedure

To prepare your RHOSP environment for the minor update, complete the following procedures:

1.1. Upgrade paths for long life releases

Familiarize yourself with the possible update and upgrade paths before you begin an update or an upgrade.

Note

You can view your current RHOSP and RHEL versions in the /etc/rhosp-release and /etc/redhat-release files.

Table 1.1. Updates version path

Current versionTarget version

RHOSP 10.0.x on RHEL 7.x

RHOSP 10.0 latest on RHEL 7.7 latest

RHOSP 13.0.x on RHEL 7.x

RHOSP 13.0 latest on RHEL 7.9 latest

RHOSP 16.1.x on RHEL 8.2

RHOSP 16.1 latest on RHEL 8.2 latest

RHOSP 16.1.x on RHEL 8.2

RHOSP 16.2 latest on RHEL 8.4 latest

RHOSP 16.2.x on RHEL 8.4

RHOSP 16.2 latest on RHEL 8.4 latest

Table 1.2. Upgrades version path

Current versionTarget version

RHOSP 10 on RHEL 7.7

RHOSP 13 latest on RHEL 7.9 latest

RHOSP 13 on RHEL 7.9

RHOSP 16.1 latest on RHEL 8.2 latest

RHOSP 13 on RHEL 7.9

RHOSP 16.2 latest on RHEL 8.4 latest

For more information, see Framework for Upgrades (13 to 16.2).

Red Hat provides two options for upgrading your environment to the next long life release:

In-place upgrade
Perform an upgrade of the services in your existing environment. This guide primarily focuses on this option.
Parallel migration
Create a new Red Hat OpenStack Platform 16.2 environment and migrate your workloads from your current environment to the new environment. For more information about Red Hat OpenStack Platform parallel migration, contact Red Hat Global Professional Services.
Important

The durations in this table are minimal estimates based on internal testing and might not apply to all productions environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. To accurately gauge the upgrade duration for each task, perform these procedures in a test environment with hardware similar to your production environment.

Table 1.3. Impact and duration of upgrade paths

 In-place upgradeParallel migration

Upgrade duration for undercloud

Estimated duration for each major action includes the following:

  • 30 minutes for Leapp upgrade command
  • 30 minutes for Leapp reboot
  • 40 minutes for director upgrade

None. You are creating a new undercloud in addition to your existing undercloud.

Upgrade duration for overcloud control plane

Estimates for each Controller node:

  • 60 minutes for Leapp upgrade and reboot
  • 60 minutes for service upgrade

None. You are creating a new control plane in addition to your existing control plane.

Outage duration for control plane

The duration of the service upgrade of the bootstrap Controller node, which is approximately 60 minutes.

None. Both overclouds are operational during the workload migration.

Consequences of control plane outage

You cannot perform OpenStack operations during the outage.

No outage.

Upgrade duration for overcloud data plane

Estimates for each Compute node and Ceph Storage node:

  • 60 minutes for Leapp upgrade and reboot
  • 30 minutes for service upgrade

None. You are creating a new data plane in addition to your existing data plane.

Outage duration for data plane

The outage is minimal due to workload migration from node to node.

The outage is minimal due to workload migration from overcloud to overcloud.

Additional hardware requirements

No additional hardware is required.

Additional hardware is required to create a new undercloud and overcloud.

1.2. Locking the environment to a Red Hat Enterprise Linux release

Red Hat OpenStack Platform (RHOSP) 16.2 is supported on Red Hat Enterprise Linux (RHEL) 8.4. Before you perform the update, lock the undercloud and overcloud repositories to the RHEL 8.4 release to avoid upgrading the operating system to a newer minor release.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check if your subscription management configuration includes the rhsm_release parameter. If the rhsm_release parameter is not present, add it and set it to 8.4:

    parameter_defaults:
      RhsmVars:
        …​
        rhsm_username: "myusername"
        rhsm_password: "p@55w0rd!"
        rhsm_org_id: "1234567"
        rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
        rhsm_method: "portal"
        rhsm_release: "8.4"
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to lock the operating system version to RHEL 8.4 on all nodes:

    $ cat > ~/set_release.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: set release to 8.4
          command: subscription-manager release --set=8.4
          become: true
    EOF
  8. Run the set_release.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/set_release.yaml --limit <undercloud>,<Controller>,<Compute>
    • Use the --limit option to apply the content to all RHOSP nodes. Replace <undercloud>, <Controller>, <Compute> with the Ansible groups in your environment that contain those nodes.
    • You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.
Note

To manually lock a node to a version, log in to the node and run the subscription-manager release command:

$ sudo subscription-manager release --set=8.4

1.3. Switching to TUS repositories

Your Red Hat OpenStack Platform (RHOSP) subscription includes repositories for Red Hat Enterprise Linux (RHEL) 8.4 Extended Update Support (EUS), in addition to standard repositories. After May 30, 2023, you must enable the RHEL 8.4 Telecommunications Update Service (TUS) repositories for Maintenance Support. The TUS repositories include the latest security patches and bug fixes for RHEL 8.4.

Switch your repositories to the required TUS repositories before you perform an update.

Table 1.4. Switching from EUS repositories to TUS repositories

EUS repositoryTUS repository

rhel-8-for-x86_64-baseos-eus-rpms

rhel-8-for-x86_64-baseos-tus-rpms

rhel-8-for-x86_64-appstream-eus-rpms

rhel-8-for-x86_64-appstream-tus-rpms

rhel-8-for-x86_64-highavailability-eus-rpms

rhel-8-for-x86_64-highavailability-tus-rpms

Table 1.5. Switching from standard repositories to TUS repositories

Standard repositoryTUS repository

rhel-8-for-x86_64-baseos-rpms

rhel-8-for-x86_64-baseos-tus-rpms

rhel-8-for-x86_64-appstream-rpms

rhel-8-for-x86_64-appstream-tus-rpms

rhel-8-for-x86_64-highavailability-rpms

rhel-8-for-x86_64-highavailability-tus-rpms

Important

You must use TUS repositories to retain compatibility with a specific version of Podman. Later versions of Podman are untested with Red Hat OpenStack Platform 16.2 and can cause unexpected results.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check the rhsm_repos parameter in your subscription management configuration. If this parameter does not include the TUS repositories, change the relevant repositories to the TUS versions:

    parameter_defaults:
      RhsmVars:
        rhsm_repos:
          - rhel-8-for-x86_64-baseos-tus-rpms
          - rhel-8-for-x86_64-appstream-tus-rpms
          - rhel-8-for-x86_64-highavailability-tus-rpms
          - ansible-2.9-for-rhel-8-x86_64-rpms
          - openstack-16.2-for-rhel-8-x86_64-rpms
          - rhceph-4-tools-for-rhel-8-x86_64-rpms
          - fast-datapath-for-rhel-8-x86_64-rpms
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to set the repositories to RHEL 8.4 TUS on all nodes:

    $ cat > ~/change_tus.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change to tus repos
          command: subscription-manager repos --disable=rhel-8-for-x86_64-baseos-eus-rpms --disable=rhel-8-for-x86_64-appstream-eus-rpms --disable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms
          become: true
    EOF
    • If your environment includes standard repositories, disable the following repositories:

      • rhel-8-for-x86_64-baseos-rpms
      • rhel-8-for-x86_64-appstream-rpms
      • rhel-8-for-x86_64-highavailability-rpms
  8. Run the change_tus.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/change_tus.yaml --limit <undercloud>,<Controller>,<Compute>
    • Use the --limit option to apply the content to all Red Hat OpenStack Platform nodes. Replace <undercloud>, <Controller>, <Compute> with the Ansible groups in your environment that contain those nodes.
    • You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.

1.4. Updating Red Hat Openstack Platform and Ansible repositories

Update your repositories to use Red Hat OpenStack Platform (RHOSP) 16.2 and Ansible 2.9 packages.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check the rhsm_repos parameter in your subscription management configuration. If the rhsm_repos parameter uses the RHOSP 16.1 and Ansible 2.8 repositories, change the repository to the correct versions:

    parameter_defaults:
      RhsmVars:
        rhsm_repos:
          - rhel-8-for-x86_64-baseos-tus-rpms
          - rhel-8-for-x86_64-appstream-tus-rpms
          - rhel-8-for-x86_64-highavailability-tus-rpms
          - ansible-2.9-for-rhel-8-x86_64-rpms
          - openstack-16.2-for-rhel-8-x86_64-rpms
          - fast-datapath-for-rhel-8-x86_64-rpms
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to set the repositories to RHOSP 16.2 on all RHOSP nodes:

    $ cat > ~/update_rhosp_repos.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change osp repos
          command: subscription-manager repos --disable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms
          become: true
    EOF
  8. Run the update_rhosp_repos.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_rhosp_repos.yaml --limit <undercloud>,<Controller>,<Compute>
    • Use the --limit option to apply the content to all RHOSP nodes. Replace <undercloud>, <Controller>, <Compute> with the Ansible groups in your environment that contain those nodes.
    • You cannot run this playbook against Ceph Storage nodes if you are using a different subscription for these nodes.
  9. Create a playbook that contains a task to set the repositories to RHOSP 16.2 on all Ceph Storage nodes:

    $ cat > ~/update_ceph_repos.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change ceph repos
          command: subscription-manager repos --disable=openstack-16-deployment-tools-for-rhel-8-x86_64-rpms --enable=openstack-16.2-deployment-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms
          become: true
    EOF
  10. Run the update_ceph_repos.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_ceph_repos.yaml --limit CephStorage

    Use the --limit option to apply the content to Ceph Storage nodes.

1.5. Setting the container-tools module version

Set the container-tools module to version 3.0 to ensure that you use the correct package versions on all nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  4. Create a playbook that contains a task to set the container-tools module to version 3.0 on all nodes:

    $ cat > ~/container-tools.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: disable default dnf module for container-tools
          command: dnf module reset -y container-tools
          become: true
        - name: set dnf module for container-tools:3.0
          command: dnf module enable -y container-tools:3.0
          become: true
    EOF
  5. Run the container-tools.yaml playbook against all nodes:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/container-tools.yaml

1.6. Updating the container image preparation file

The container preparation file is the file that contains the ContainerImagePrepare parameter. You use this file to define the rules for obtaining container images for the undercloud and overcloud.

Before you update your environment, check the file to ensure that you obtain the correct image versions.

Procedure

  1. Edit the container preparation file. The default name for this file is usually containers-prepare-parameter.yaml.
  2. Check the tag parameter is set to 16.2 for each rule set:

    parameter_defaults:
      ContainerImagePrepare:
      - push_destination: true
        set:
          …​
          tag: '16.2'
        tag_from_label: '{version}-{release}'
    Note

    If you do not want to use a specific tag for the update, such as 16.2 or 16.2.2, remove the tag key-value pair and specify tag_from_label only. This uses the installed Red Hat OpenStack Platform version to determine the value for the tag to use as part of the update process.

  3. Save this file.

1.7. Updating the SSL/TLS configuration

Remove the NodeTLSData resource from the resource_registry to update your SSL/TLS configuration.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your custom overcloud SSL/TLS public endpoint file, which is usually named ~/templates/enable-tls.yaml.
  4. Remove the NodeTLSData resource from the resource_registry:

    resource_registry:
      OS::TripleO::NodeTLSData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/tls-cert-inject.yaml
      ...

    The overcloud deployment uses a new service in HAProxy to determine if SSL/TLS is enabled.

    Note

    If this is the only resource in the resource_registry section of the enable-tls.yaml file, remove the complete resource_registry section.

  5. Save the SSL/TLS public endpoint file.
  6. If you are updating from Red Hat OpenStack Platform 16.1, you must update the permissions in Red Hat Identity Manager (IdM) for all pre-update checks to pass. Use ssh to login to the server that is running your IdM, and then run the following commands:

    $ kinit admin
    $ ipa privilege-add-permission 'Nova Host Management' --permission 'System: Modify Realm Domains'

1.8. Disabling fencing in the overcloud

Before you update the overcloud, ensure that fencing is disabled.

If fencing is deployed in your environment during the Controller nodes update process, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results.

If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the update to avoid any unintended results.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file.

    $ source ~/stackrc
  3. Log in to a Controller node and run the Pacemaker command to disable fencing:

    $ ssh heat-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"

    Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command.

  4. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the update process.