Framework for Upgrades (13 to 16.1)

Red Hat OpenStack Platform 16.1

In-place upgrades from Red Hat OpenStack Platform 13 to 16.1

OpenStack Documentation Team

Abstract

This guide contains information on the framework for the in-place upgrades across long-life versions. This framework includes tools to upgrade your OpenStack Platform environment from one long life version to the next long life version. In this case, the guide focuses on upgrading from Red Hat OpenStack Platform 13 (Queens) to 16.1 (Train).

Chapter 1. About the Red Hat OpenStack Platform framework for upgrades

The Red Hat OpenStack Platform framework for upgrades is a workflow to upgrade your Red Hat OpenStack Platform environment from one long life version to the next long life version. This workflow is an in-place solution and the upgrade occurs within your existing environment.

1.1. Upgrade framework for long life versions

You can use the Red Hat OpenStack Platform upgrade framework to perform an in-place upgrade path through multiple versions of the overcloud. The goal is to provide you with an opportunity to remain on certain OpenStack versions that are considered long life versions and upgrade when the next long life version is available.

This guide provides an upgrade framework through the following versions:

Current VersionTarget Version

Red Hat OpenStack Platform 13

Red Hat OpenStack Platform 16.1

1.2. Lifecycle support for long life versions

ReleaseSupport BeginsProduction Support EndsELS Support Ends

13

Jun 2018

Full: Jan 2020 (18 months)
Maintenance: Jun 2021 (36 months)

Jun 2023 (24 months)

16.0

Feb 2020

Oct 2020 (8 months)

 

16.1

Jul 2020

Apr 2023 (34 months)

 

1.3. Upgrade paths for long life release

Red Hat provides two options for upgrading your environment to the next long life release:

In-place upgrade
Perform an upgrade of the services in your existing environment. This guide primarily focuses on this option.
Parallel migration
Create a new Red Hat OpenStack Platform 16.1 environment and migrate your workloads from your current environment to the new environment. For more information about Red Hat OpenStack Platform parallel migration, contact Red Hat Global Professional Services.
Important

The durations in this table are minimal estimates based on internal testing and might not apply to all productions environments. For example, if your hardware has low specifications or an extended boot period, allow for more time with these durations. To accurately gauge the upgrade duration for each task, perform these procedures in a test environment with hardware similar to your production environment.

Table 1.1. Impact and duration of upgrade paths

 In-place upgradeParallel migration

Upgrade duration for undercloud

Estimated duration for each major action includes the following:

  • 30 minutes for Leapp upgrade command
  • 30 minutes for Leapp reboot
  • 40 minutes for director upgrade

None. You are creating a new undercloud in addition to your existing undercloud.

Upgrade duration for overcloud control plane

Estimates for each Controller node:

  • 60 minutes for Leapp upgrade and reboot
  • 60 minutes for service upgrade

None. You are creating a new control plane in addition to your existing control plane.

Outage duration for control plane

The duration of the service upgrade of the bootstrap Controller node, which is approximately 60 minutes.

None. Both overclouds are operational during the workload migration.

Consequences of control plane outage

You cannot perform OpenStack operations during the outage.

No outage.

Upgrade duration for overcloud data plane

Estimates for each Compute node and Ceph Storage node:

  • 60 minutes for Leapp upgrade and reboot
  • 30 minutes for service upgrade

None. You are creating a new data plane in addition to your existing data plane.

Outage duration for data plane

The outage is minimal due to workload migration from node to node.

The outage is minimal due to workload migration from overcloud to overcloud.

Additional hardware requirements

No additional hardware is required.

Additional hardware is required to create a new undercloud and overcloud.

Chapter 2. Planning and preparation for an in-place upgrade

Before you conduct an in-place upgrade of your OpenStack Platform environment, create a plan for the upgrade and accommodate any potential obstacles that might block a successful upgrade.

2.1. Familiarize yourself with Red Hat OpenStack Platform 16.1

Before you perform an upgrade, familiarize yourself with Red Hat OpenStack Platform 16.1 to help you understand the resulting environment and any potential version-to-version changes what might affect your upgrade. To familiarize yourself with Red Hat OpenStack Platform 16.1, follow these suggestions:

2.2. High level changes in Red Hat OpenStack Platform 16.1

The following high-level changes occur during the upgrade to Red Hat OpenStack Platform 16.1:

  • OpenStack Platform director 16.1 configures the overcloud using an Ansible-driven method called config-download. This replaces the standard heat-based configuration method. Director still uses heat to orchestrate provisioning operations.
  • The director installation uses the same method as the overcloud deployment. Therefore, the undercloud also uses openstack-tripleo-heat-templates as a blueprint for installing and configuring each service.
  • The undercloud runs OpenStack services in containers.
  • The undercloud pulls and stores container images through a new method. Instead of pulling container images before deploying the overcloud, the undercloud pulls all relevant container images during the deployment process.
  • The overcloud deployment process includes an Advanced Subscription Management method to register nodes. This method incorporates an Ansible role to register OpenStack Platform nodes. The new method also applies different subscriptions to different node roles if necessary.
  • The overcloud now uses Open Virtual Network (OVN) as the default ML2 mechanism driver. It is possible to migrate your Open vSwitch (OVS) service to OVN, which you perform after the completion of a successful upgrade.
  • The undercloud and overcloud both run on Red Hat Enterprise Linux 8.
  • openstack-tripleo-heat-templates includes a unified composable service template collection in the deployment directory. This directory now includes templates with merged content from both the containerized service and Puppet-based composable service templates.
  • The OpenStack Data Processing service (sahara) is no longer supported.

    Important

    If you have sahara enabled in your Red Hat OpenStack Platform 13 environment, do not continue with this upgrade and contact Red Hat Global Support Services.

  • The OpenStack Telemetry components are deprecated in favor of the Service Telemetry Framework (STF).

2.3. Changes in Red Hat Enterprise Linux 8

The undercloud and overcloud both run on Red Hat Enterprise Linux 8. This includes new tools and functions relevant to the undercloud and overcloud:

  • The undercloud and overcloud use the Red Hat Container Toolkit. Instead of docker to build and control the container lifecycle, Red Hat Enterprise Linux 8 includes buildah to build new container images and podman for container management.
  • Red Hat Enterprise Linux 8 does not include the docker-distribution package. The undercloud now includes a private HTTP registry to provide container images to overcloud nodes.
  • The upgrade process from Red Hat Enterprise Linux 7 to 8 uses the leapp tool.
  • Red Hat Enterprise Linux 8 does not use the ntp service. Instead, Red Hat Enterprise Linux 8 uses chronyd.
  • Red Hat Enterprise Linux 8 includes new versions of high availability tools.

The Red Hat OpenStack Platform 16.1 uses Red Hat Enterprise Linux 8.2 as the base operating system. As a part of the upgrade process, you will upgrade the base operating system of nodes to Red Hat Enterprise Linux 8.2.

For more information about the key differences between Red Hat Enterprise Linux 7 and 8, see "Considerations in adopting RHEL 8". For general information about Red Hat Enterprise linux 8, see Product Documentation for Red Hat Enterprise Linux 8.

2.4. Leapp upgrade usage in Red Hat OpenStack Platform

The long-life Red Hat OpenStack Platform upgrade also requires an upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8. Red Hat Enterprise Linux 7 includes a tool named leapp, which performs the upgrade to Red Hat Enterprise Linux 8. Both the undercloud and overcloud use a separate process for performing the operating system upgrade.

Undercloud process

Run the leapp upgrade manually before you run the openstack undercloud upgrade command. The undercloud upgrade includes instructions for performing the leapp upgrade.

Overcloud process

The overcloud upgrade framework automatically runs the leapp upgrade.

Limitations

For information of potential limitations that might affect your upgrade, see the following sections from the Upgrading from RHEL 7 to RHEL 8 guide:

If any known limitations affect your environment, seek advice from Red Hat Technical Support Team.

2.5. Known issues that might block an upgrade

Review the following known issues that might affect a successful upgrade.

Upgrade Framework Issues

BZ#1851914 - system_upgrade reports an error after Leapp upgrade

The openstack overcloud upgrade run --tags system_upgrade command runs a Leapp upgrade on an overcloud node. After the upgrade and reboot, an Ansible error occurs during the deployment steps:

"No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"

This error does not impact the remaining upgrade steps. The procedures in this guide include a workaround for this command to run only the upgrade_steps_playbook.yaml and not the additional deployment steps:

$ openstack overcloud upgrade run --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

Red Hat aims to resolve this issue in the next 16.1 minor release update.

BZ#1849235 - Unset UpgradeLevelNovaCompute or live migrations fail

The UpgradeLevelNovaCompute parameter is set to auto which causes OpenStack Compute (nova) to report an error

nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy

This guide includes a procedure to explicitly set this parameter to an empty value (''). Red Hat aims to resolve this issue in the next 16.1 minor release update.

BZ#1852360 - swift_rsync and swift_rsync_healthcheck moved in failed state after undercloud upgrade
The containerized tripleo_swift_rsync service on the Red Hat OpenStack Platform 16.1 undercloud reports a failed state. This is due to a conflict with the Red Hat OpenStack Platform 16.1 version running the OpenStack Object Storage (swift) rsync daemon through xinetd. This guide includes xinetd in the list of packages that you must remove remove before to the undercloud upgrade. Red Hat aims to resolve this issue in the next 16.1 minor release update.
BZ#1845726 - Automation of Leapp data file installation on the overcloud
This guide provides a step to copy the Leapp data files from the undercloud to the overcloud nodes. Red Hat aims to automate this procedure in the next 16.1 minor release update.

NFV and iSCSI Storage Issues

BZ#1853433 - To allow system_upgrade step, we need to unmount NFS / iSCSI filesystems in the overcloud nodes (and clear fstab)
The Leapp upgrade fails with any NFS shares mounted. Specifically, the nodes that run the Compute Service (nova) or the Image Service (glance) services hang if they used an NFS mount. Red Hat aims to resolve this issue in the next 16.1 minor release update.

Red Hat Ceph Storage Issues

BZ#1855813 - Ceph tools repository should be switched from RHCS3 to RHCS4 only after converge, before running external-upgrade
The ceph-ansible playbook collection on the undercloud deploys containers on the overcloud. To upgrade your environment, you must have Red Hat Ceph Storage 3 version of ceph-ansible to maintain Ceph Storage 3 containers through the upgrade. The Red Hat Ceph Storage 3 version of ceph-ansible is not available on the undercloud after the upgrade to Red Hat Enterprise Linux 8. Red Hat is currently developing an alternative process to retain ceph-ansible for Red Hat Ceph Storage 3 on Red Hat Enterprise Linux 8.
BZ#1853457 - RGW does not come back up after docker_to_podman
The containers for Ceph Object Gateway (RGW) fail to start after an upgrade. Red Hat aims to resolve this issue in the next ceph-ansible minor release update.
BZ#1844591 - Playbook docker-to-podman does not support ceph-disk deployed environments
The playbook that changes the systemd unit files from docker to podman does not support Ceph Storage customers using ceph-disk methods. If the osd_scenario option in your CephAnsibleDisksConfig heat parameter is set to collocated or non-collocated, this issue affects your environment. This issue does not affect environments using ceph-volume methods, such as the osd_scenario set to lvm. Red Hat aims to resolve this issue in the next ceph-ansible minor release update.
BZ#1853275 - Set noout,norebalance flags on the cluster before LEAPP reboots the node
This guide includes commands to set noout and norebalance flags prior to any Ceph Storage OSD node upgrade. Red Hat aims to automate this procedure in the next 16.1 minor release update.
BZ#1850991 - Upgrade fails when checking for HAProxy Red Hat Ceph Storage dashboard listener status
Director configures the Red Hat Ceph Storage dashboard listener in the HAProxy configuration even if the dashboard is disabled. As a result, upgrades of Red Hat OpenStack Platform with a Ceph Storage cluster fail. Red Hat aims to resolve this issue in the next 16.1 minor release update.

2.6. Backup and restore

Before you upgrade your Red Hat OpenStack Platform 13 environment, back up the undercloud and overcloud control plane. For more information about backing up nodes with the Relax-and-recover (ReaR) utility, see the Undercloud and Control Plane Back Up and Restore guide.

2.7. Minor version update

Before you upgrade your Red Hat OpenStack Platform environment, update the environment to the latest minor version of your current release. For example, perform an update of your Red Hat OpenStack Platform 13 environment to the latest 13 before running the upgrade to Red Hat OpenStack Platform 16.1

For instructions on performing a minor version update for Red Hat OpenStack Platform 13, see "Keeping Red Hat OpenStack Platform Updated".

2.8. Validating Red Hat OpenStack Platform 13 before the upgrade

Before you upgrade to Red Hat OpenStack Platform 16.1, validate your undercloud and overcloud with the tripleo-validations playbooks. In Red Hat OpenStack Platform 13, you run these playbooks through the OpenStack Workflow Service (mistral).

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create a bash script called pre-upgrade-validations.sh and include the following content in the script:

    #!/bin/bash
    for VALIDATION in $(openstack action execution run tripleo.validations.list_validations '{"groups": ["pre-upgrade"]}' | jq ".result[] | .id")
    do
      echo "=== Running validation: $VALIDATION ==="
      ID=$(openstack workflow execution create -f value -c ID tripleo.validations.v1.run_validation "{\"validation_name\": $VALIDATION}")
      while [ $(openstack workflow execution show $ID -f value -c State) == "RUNNING" ]
      do
        sleep 1
      done
      echo ""
      openstack workflow execution output show $ID | jq -r ".stdout"
      echo ""
    done
  3. Add permission to run the script:

    $ chmod +x pre-upgrade-validations.sh
  4. Run the script:

    $ ./pre-upgrade-validations.sh

    Review the script output to determine which validations succeed and fail:

    === Running validation: "check-ftype" ===
    
    Success! The validation passed for all hosts:
    * undercloud

Chapter 3. Repositories

This section contains the repositories for the undercloud and overcloud. Refer to this section when you need to enable repositories is certain situations:

  • Enabling repositories when registering to the Red Hat Customer Portal.
  • Enabling and synchronizing repositories to your Red Hat Satellite server.
  • Enabling repositories when registering to your Red hat Satellite server.

3.1. Undercloud repositories

Red Hat OpenStack Platform 16.1 runs on Red Hat Enterprise Linux 8.2. As a result, you must lock the content from these repositories to the respective Red Hat Enterprise Linux version.

Warning

Any repositories outside the ones specified here are not supported. Ensure that you do not enable Extra Packages for Enterprise Linux (EPEL) or the upgrade will fail.

Core repositories

The following table lists core repositories for installing the undercloud.

NameRepositoryDescription of requirement

Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS)

rhel-8-for-x86_64-baseos-eus-rpms

Base operating system repository for x86_64 systems.

Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)

rhel-8-for-x86_64-appstream-eus-rpms

Contains Red Hat OpenStack Platform dependencies.

Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs) Extended Update Support (EUS)

rhel-8-for-x86_64-highavailability-eus-rpms

High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability.

Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs)

ansible-2.9-for-rhel-8-x86_64-rpms

Ansible Engine for Red Hat Enterprise Linux. Used to provide the latest version of Ansible.

Red Hat Satellite Tools for RHEL 8 Server RPMs x86_64

satellite-tools-6.5-for-rhel-8-x86_64-rpms

Tools for managing hosts with Red Hat Satellite 6.

Red Hat OpenStack Platform 16.1 for RHEL 8 (RPMs)

openstack-16.1-for-rhel-8-x86_64-rpms

Core Red Hat OpenStack Platform repository, which contains packages for Red Hat OpenStack Platform director.

Red Hat Fast Datapath for RHEL 8 (RPMS)

fast-datapath-for-rhel-8-x86_64-rpms

Provides Open vSwitch (OVS) packages for OpenStack Platform.

Ceph repositories

The following table lists Ceph Storage related repositories for the undercloud.

NameRepositoryDescription of Requirement

Red Hat Ceph Storage Tools 4 for RHEL 8 x86_64 (RPMs)

rhceph-4-tools-for-rhel-8-x86_64-rpms

Provides tools for nodes to communicate with the Ceph Storage cluster. The undercloud requires the ceph-ansible package from this repository if you plan to use Ceph Storage in your overcloud.

IBM POWER repositories

The following table contains a list of repositories for Red Hat Openstack Platform on POWER PC architecture. Use these repositories in place of equivalents in the Core repositories.

NameRepositoryDescription of requirement

Red Hat Enterprise Linux for IBM Power, little endian - BaseOS (RPMs)

rhel-8-for-ppc64le-baseos-rpms

Base operating system repository for ppc64le systems.

Red Hat Enterprise Linux 8 for IBM Power, little endian - AppStream (RPMs)

rhel-8-for-ppc64le-appstream-rpms

Contains Red Hat OpenStack Platform dependencies.

Red Hat Enterprise Linux 8 for IBM Power, little endian - High Availability (RPMs)

rhel-8-for-ppc64le-highavailability-rpms

High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability.

Red Hat Ansible Engine 2.8 for RHEL 8 IBM Power, little endian (RPMs)

ansible-2.8-for-rhel-8-ppc64le-rpms

Ansible Engine for Red Hat Enterprise Linux. Provides the latest version of Ansible.

Red Hat OpenStack Platform 16.1 for RHEL 8 (RPMs)

openstack-16.1-for-rhel-8-ppc64le-rpms

Core Red Hat OpenStack Platform repository for ppc64le systems.

3.2. Overcloud repositories

Red Hat OpenStack Platform 16.1 runs on Red Hat Enterprise Linux 8.2. As a result, you must lock the content from these repositories to the respective Red Hat Enterprise Linux version.

Warning

Any repositories outside the ones specified here are not supported. Ensure that you do not enable Extra Packages for Enterprise Linux (EPEL) or the upgrade will fail.

Core repositories

The following table lists core repositories for installing the overcloud.

NameRepositoryDescription of requirement

Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) Extended Update Support (EUS)

rhel-8-for-x86_64-baseos-eus-rpms

Base operating system repository for x86_64 systems.

Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)

rhel-8-for-x86_64-appstream-eus-rpms

Contains Red Hat OpenStack Platform dependencies.

Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs) Extended Update Support (EUS)

rhel-8-for-x86_64-highavailability-eus-rpms

High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability.

Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs)

ansible-2.9-for-rhel-8-x86_64-rpms

Ansible Engine for Red Hat Enterprise Linux. Used to provide the latest version of Ansible.

Advanced Virtualization for RHEL 8 x86_64 (RPMs)

advanced-virt-for-rhel-8-x86_64-rpms

Provides virtualization packages for OpenStack Platform.

Red Hat Satellite Tools for RHEL 8 Server RPMs x86_64

satellite-tools-6.5-for-rhel-8-x86_64-rpms

Tools for managing hosts with Red Hat Satellite 6.

Red Hat OpenStack Platform 16.1 for RHEL 8 (RPMs)

openstack-16.1-for-rhel-8-x86_64-rpms

Core Red Hat OpenStack Platform repository.

Red Hat Fast Datapath for RHEL 8 (RPMS)

fast-datapath-for-rhel-8-x86_64-rpms

Provides Open vSwitch (OVS) packages for OpenStack Platform.

Real Time repositories

The following table lists repositories for Real Time Compute (RTC) functionality.

NameRepositoryDescription of requirement

Red Hat Enterprise Linux 8 for x86_64 - Real Time (RPMs)

rhel-8-for-x86_64-rt-rpms

Repository for Real Time KVM (RT-KVM). Contains packages to enable the real time kernel. Enable this repository for all Compute nodes targeted for RT-KVM. NOTE: You need a separate subscription to a Red Hat OpenStack Platform for Real Time SKU to access this repository.

Red Hat Enterprise Linux 8 for x86_64 - Real Time for NFV (RPMs)

rhel-8-for-x86_64-nfv-rpms

Repository for Real Time KVM (RT-KVM) for NFV. Contains packages to enable the real time kernel. Enable this repository for all NFV Compute nodes targeted for RT-KVM. NOTE: You need a separate subscription to a Red Hat OpenStack Platform for Real Time SKU to access this repository.

IBM POWER repositories

The following table lists repositories for Openstack Platform on POWER PC architecture. Use these repositories in place of equivalents in the Core repositories.

NameRepositoryDescription of requirement

Red Hat Enterprise Linux for IBM Power, little endian - BaseOS (RPMs)

rhel-8-for-ppc64le-baseos-rpms

Base operating system repository for ppc64le systems.

Red Hat Enterprise Linux 8 for IBM Power, little endian - AppStream (RPMs)

rhel-8-for-ppc64le-appstream-rpms

Contains Red Hat OpenStack Platform dependencies.

Red Hat Enterprise Linux 8 for IBM Power, little endian - High Availability (RPMs)

rhel-8-for-ppc64le-highavailability-rpms

High availability tools for Red Hat Enterprise Linux. Used for Controller node high availability.

Red Hat Ansible Engine 2.8 for RHEL 8 IBM Power, little endian (RPMs)

ansible-2.8-for-rhel-8-ppc64le-rpms

Ansible Engine for Red Hat Enterprise Linux. Used to provide the latest version of Ansible.

Red Hat OpenStack Platform 16.1 for RHEL 8 (RPMs)

openstack-16.1-for-rhel-8-ppc64le-rpms

Core Red Hat OpenStack Platform repository for ppc64le systems.

3.3. Red Hat Satellite 6 considerations

If you use Red Hat Satellite 6 to host RPMs and container images for your Red Hat OpenStack Platform environment, you must account for certain considerations when you use Satellite 6 to deliver content during the Red Hat OpenStack Platform 16.1 upgrade.

Assumptions about your current environment

  • Your Satellite server already hosts Red Hat OpenStack Platform 13 RPMs and container images.
  • You have already registered all nodes in your Red Hat OpenStack Platform 13 environment to the Satellite server. For example, you previously used an activation key linked to an Red Hat OpenStack Platform 13 content view to register nodes to OpenStack Platform 13 content.

Recommendations for Red Hat OpenStack Platform upgrades

  • Enable and synchronize the necessary RPM repositories for both the Red Hat OpenStack Platform 13 undercloud and overcloud. This includes the necessary Red Hat Enterprise Linux 8.2 repositories.
  • Create custom products on the Satellite server to host container images for the following Red Hat OpenStak Platform versions:

    • Red Hat OpenStack Platform 16.1
    • Red Hat OpenStack Platform 15
  • Create and promote a content view for Red Hat OpenStack Platform 16.1 upgrade and include the following content in the content view:

    • All undercloud and overcloud RPM repositories, including Red Hat Enterprise Linux 8.2 repositories.
    • Red Hat OpenStack Platform 16.1 container images.
    • Red Hat OpenStack Platform 15 container images.
  • Create an activation key for the upgrade to Red Hat OpenStack Platform 16.1 and associate the activation key with the Red Hat OpenStack Platform 16.1 content view. Pin this activation key to Red Hat Enterprise Linux 8.2.
  • Check that no node has the katello-host-tools-fact-plugin package installed. The Leapp upgrade does not upgrade this package and leaving this package on a Red Hat Enterprise Linux 8.2 system causes subscription-manager to report errors.

Part I. Upgrading the undercloud

Chapter 4. Preparing for the undercloud upgrade

Before you perform the undercloud upgrade, you must complete some preparation steps so that the undercloud upgrade runs successfully.

4.1. New memory requirements

In Red Hat OpenStack Platform 16.1, the undercloud has new memory requirements:

Red Hat OpenStack Platform 13Red Hat OpenStack Platform 16.1

16 GB RAM

24 GB RAM

Ensure that your undercloud meets these new requirements before you proceed with the upgrade.

4.2. Using predictable NIC names for the undercloud node

Before you run the Leapp upgrade on the undercloud node, you must check for kernel-based NIC names, which usually contain a eth prefix. These NIC names are usually unpredictable in terms of NIC assignments. The Leapp upgrade fails if any NICs on the node use the eth prefix.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create an Ansible playbook named playbook-nics.yaml and copy the following content into the playbook:

    ---
    - name: Rename eth devices
      hosts: all
      become: yes
      vars:
        prefix: "em"
      tasks:
        - name: Update udev rules
          lineinfile:
            line: "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"{{ ansible_facts[item]['perm_macaddress'] | default(ansible_facts[item]['macaddress']) }}\", NAME=\"{{ prefix }}{{ item|replace('eth','') }}\""
            path: /etc/udev/rules.d/70-rhosp-persistent-net.rules
            create: True
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
        - name: Copy network-script files using the new prefix
          copy:
            remote_src: True
            src: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
            dest: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
        - name: Edit NAME in new network-script files
          lineinfile:
            regexp: "^NAME=.*"
            line: "NAME={{ prefix }}{{ item|replace('eth','') }}"
            path: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
        - name: Edit DEVICE in new network-script files
          lineinfile:
            regexp: "^DEVICE=.*"
            line: "DEVICE={{ prefix }}{{ item|replace('eth','') }}"
            path: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
        - name: Backup old eth network-script files
          copy:
            remote_src: True
            src: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
            dest: /etc/sysconfig/network-scripts/ifcfg-{{ item }}.bak
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
        - name: Backup old eth network-script files
          file:
            path: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
            state: absent
          with_items: "{{ ansible_interfaces }}"
          when: item.startswith("eth")
    Note

    You will use this playbook to rename the overcloud NICs at a later stage in the upgrade process.

  3. Run the playbook-nics.yaml playbook on the undercloud:

    $ ansible-playbook -c local -i localhost, playbook-nics.yaml

    The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

    $ ansible-playbook -c local -i localhost, -e prefix="mynic" ~/playbook-nics.yaml
  4. Reboot the undercloud node using the standard reboot procedures. For more information, see "Rebooting nodes".

4.3. Setting the SSH root permission parameter on the undercloud

The Leapp upgrade checks whether the PermitRootLogin parameter exists in the /etc/ssh/sshd_config file. You must explicitly set this parameter to either yes or no.

For security purposes, set this parameter to no to disable SSH access to the root user on the undercloud.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

    $ sudo grep PermitRootLogin /etc/ssh/sshd_config
  3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

    PermitRootLogin no
  4. Save the file.

Chapter 5. Upgrading the undercloud operating system

Before you upgrade director, you must upgrade the undercloud operating system from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8. As a part of this operating system upgrade, you must remove the Red Hat OpenStack Platform 13 packages and then run the Leapp utility to upgrade the system packages. This package removal and operating system upgrade does not affect the undercloud database. After you complete the operating system upgrade, reinstall the Red Hat OpenStack Platform 16.1 versions of the director packages.

5.1. Removing Red Hat OpenStack Platform director packages

Before you run the Leapp utility, remove the Red Hat OpenStack Platform 13 packages tied to Red Hat Enterprise Linux 7. These package names use a release suffix of el7ost. Some el7ost remain on the system as dependencies for subscription-manager and the Leapp utility.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Disable the main OpenStack services on the undercloud:

    $ sudo systemctl stop openstack-* httpd haproxy mariadb rabbitmq* docker xinetd
  3. Remove the main OpenStack services from the undercloud, except OpenvSwitch and certain Python 2 packages that are required for the upgrade:

    $ sudo yum -y remove *el7ost* galera* haproxy* \
        httpd mysql* pacemaker* xinetd python-jsonpointer \
        qemu-kvm-common-rhev qemu-img-rhev rabbit* \
        redis* \
        -- \
        -*openvswitch* -python-docker -python-PyMySQL \
        -python-pysocks -python2-asn1crypto -python2-babel \
        -python2-cffi -python2-cryptography -python2-dateutil \
        -python2-idna -python2-ipaddress -python2-jinja2 \
        -python2-jsonpatch -python2-markupsafe -python2-pyOpenSSL \
        -python2-requests -python2-six -python2-urllib3
  4. Remove the content from the /etc/httpd and /var/lib/docker directories:

    $ sudo rm -rf /etc/httpd /var/lib/docker

5.2. Performing a Leapp upgrade on the undercloud

Install and run the Leapp utility to upgrade the operating system to Red Hat Enterprise Linux 8.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Install the Leapp utility:

    $ sudo yum install leapp
  3. Download the additional required data files (RPM package changes and RPM repository mapping) attached to the Knowledge Base article "Data required by the Leapp utility for an in-place upgrade from RHEL 7 to RHEL 8" and place these files in the /etc/leapp/files/ directory.
  4. Update your Red Hat subscription:

    • If your undercloud uses the Red Hat Customer Portal for registration, refresh your current subscription to obtain access to the Red Hat Enterprise Linux 8.2 content:

      $ sudo subscription-manager refresh
    • If your undercloud uses a Red Hat Satellite server for registration, re-register the undercloud to a content view associated with your Red Hat OpenStack Platform 16.1 activation key.

      $ subscription-manager register --force --org ORG --activationkey ACTIVATION_KEY --release 8.2
      Note

      The content view that you create for Red Hat OpenStack Platform 16.1 should contain content for Red Hat Enterprise Linux 8.2.

  5. Red Hat OpenStack Platform 16.1 uses a newer version of Open vSwitch`. Substitute the Open vSwitch version through the to_remove and to_install transaction files:

    $ echo 'openvswitch2.11' | sudo tee -a /etc/leapp/transaction/to_remove
    $ echo 'openvswitch2.13' | sudo tee -a /etc/leapp/transaction/to_install
  6. Start the Leapp upgrade process:

    $ sudo leapp upgrade --debug --enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms

    Use the --enablerepo option to set the repositories that you want to enable during the Leapp upgrade process. You must include these repositories to facilitate the Red Hat OpenStack Platform 16.1 transition, especially with the newer version of Open vSwitch.

  7. Wait for the leapp upgrade command to successfully complete.
  8. Create an empty .autorelabel file in your root directory:

    $ sudo touch /.autorelabel

    After a reboot, SELinux detects this file and automatically relabels the file system.

  9. Reboot the undercloud:

    $ sudo reboot

5.3. Removing unnecessary packages

After the Leapp upgrade, some unnecessary packages remain on the undercloud. Remove these packages.

Procedure

  1. Remove the unnecessary packages

    $ sudo dnf -y remove python2*

Chapter 6. Upgrading director

After you complete the undercloud operating system upgrade, upgrade director. The database from the previous Red Hat OpenStack Platform 13 undercloud remains on host after the operating system upgrade. Install the new Red Hat OpenStack Platform 16.1 packages and configure the new source for Red Hat OpenStack Platform 16.1 container images before you run the openstack undercloud upgrade command.

6.1. Locking the environment to a Red Hat Enterprise Linux release

Red Hat OpenStack Platform 16.1 is supported on Red Hat Enterprise Linux 8.2. Before you perform the update, you must lock the undercloud repositories to the Red Hat Enterprise Linux 8.2 release to avoid upgrading the operating system to a newer minor release.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Lock the undercloud to a specific verison with the subscription-manager release command:

    $ sudo subscription-manager release --set=8.2

6.2. Enabling repositories for the undercloud

Enable the repositories that are required for the undercloud, and update the system packages to the latest versions.

Procedure

  1. Log in to your undercloud as the stack user.
  2. Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:

    [stack@director ~]$ sudo subscription-manager repos --disable=*
    [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

    These repositories contain packages that the director installation requires.

  3. Synchronize the operating system to ensure that your system packages match the operating system version:

    [stack@director ~]$ sudo dnf distro-sync -y
    [stack@director ~]$ sudo reboot

6.3. Installing director packages

Install packages relevant to Red Hat OpenStack Platform director.

Procedure

  1. Install the command line tools for director installation and configuration:

    [stack@director ~]$ sudo dnf install -y python3-tripleoclient

6.4. Installing ceph-ansible

The ceph-ansible package is required when you use Ceph Storage with Red Hat OpenStack Platform.

If you use Red Hat Ceph Storage, or if your deployment uses an external Ceph Storage cluster, install the ceph-ansible package. If you do not plan to use Ceph Storage, do not install the ceph-ansible package.

Procedure

  1. Enable the Ceph Tools repository:

    [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
  2. Install the ceph-ansible package:

    [stack@director ~]$ sudo dnf install -y ceph-ansible

6.5. Preparing container images

The undercloud installation requires an environment file to determine where to obtain container images and how to store them. Generate and customize this environment file that you can use to prepare your container images.

Procedure

  1. Log in to your undercloud host as the stack user.
  2. Generate the default container image preparation file:

    $ openstack tripleo container image prepare default \
      --local-push-destination \
      --output-env-file containers-prepare-parameter.yaml

    This command includes the following additional options:

    • --local-push-destination sets the registry on the undercloud as the location for container images. With this option, director pulls the necessary images from the Red Hat Container Catalog and pushes the images to the registry on the undercloud. Director uses the undercloud registry as the container image source. To pull container images directly from the Red Hat Container Catalog, omit this option.
    • --output-env-file specifies an environment file that includes include the parameters for preparing your container images. In this example, the name of the file is containers-prepare-parameter.yaml.

      Note

      You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

  3. Modify the containers-prepare-parameter.yaml to suit your requirements.

6.6. Container image preparation parameters

The default file for preparing your containers (containers-prepare-parameter.yaml) contains the ContainerImagePrepare heat parameter. This parameter defines a list of strategies for preparing a set of images:

parameter_defaults:
  ContainerImagePrepare:
  - (strategy one)
  - (strategy two)
  - (strategy three)
  ...

Each strategy accepts a set of sub-parameters that defines which images to use and what to do with the images. The following table contains information about the sub-parameters you can use with each ContainerImagePrepare strategy:

ParameterDescription

excludes

List of image name substrings to exclude from a strategy.

includes

List of image name substrings to include in a strategy. At least one image name must match an existing image. All excludes are ignored if includes is specified.

modify_append_tag

String to append to the tag for the destination image. For example, if you pull an image with the tag 14.0-89 and set the modify_append_tag to -hotfix, the director tags the final image as 14.0-89-hotfix.

modify_only_with_labels

A dictionary of image labels that filter the images that you want to modify. If an image matches the labels defined, the director includes the image in the modification process.

modify_role

String of ansible role names to run during upload but before pushing the image to the destination registry.

modify_vars

Dictionary of variables to pass to modify_role.

push_destination

The namespace of the registry to push images during the upload process. When you specify a namespace for this parameter, all image parameters also use this namespace. If set to true, the push_destination is set to the undercloud registry namespace. Do not set this parameter to false in production environments. If this is set to false or not provided and the remote registry requires authentication, set the ContainerImageRegistryLogin parameter to true and provide the credentials with the ContainerImageRegistryCredentials parameter.

pull_source

The source registry from where to pull the original container images.

set

A dictionary of key: value definitions that define where to obtain the initial images.

tag_from_label

Defines the label pattern to tag the resulting images. Usually sets to {version}-{release}.

The set parameter accepts a set of key: value definitions:

KeyDescription

ceph_image

The name of the Ceph Storage container image.

ceph_namespace

The namespace of the Ceph Storage container image.

ceph_tag

The tag of the Ceph Storage container image.

name_prefix

A prefix for each OpenStack service image.

name_suffix

A suffix for each OpenStack service image.

namespace

The namespace for each OpenStack service image.

neutron_driver

The driver to use to determine which OpenStack Networking (neutron) container to use. Use a null value to set to the standard neutron-server container. Set to ovn to use OVN-based containers.

tag

The tag that the director uses to identify the images to pull from the source registry. You usually keep this key set to the default value, which is the Red Hat OpenStack Platform version number.

Note

The container images use multi-stream tags based on Red Hat OpenStack Platform version. This means there is no longer a latest tag.

The ContainerImageRegistryCredentials parameter maps a container registry to a username and password to authenticate to that registry.

If a container registry requires a username and password, you can use ContainerImageRegistryCredentials to include credentials with the following syntax:

  ContainerImagePrepare:
  - push_destination: 192.168.24.1:8787
    set:
      namespace: registry.redhat.io/...
      ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      my_username: my_password

In the example, replace my_username and my_password with your authentication credentials. Instead of using your individual user credentials, Red Hat recommends creating a registry service account and using those credentials to access registry.redhat.io content. For more information, see "Red Hat Container Registry Authentication".

The ContainerImageRegistryLogin parameter is used to control the registry login on the systems being deployed. This must be set to true if push_destination is set to false or not used.

  ContainerImagePrepare:
  - set:
      namespace: registry.redhat.io/...
      ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      my_username: my_password
  ContainerImageRegistryLogin: true

If you have configured push_destination, do not set ContainerImageRegistryLogin to true. If you set this option to true and the overcloud nodes do not have network connectivity to the registry hosts defined in ContainerImageRegistryCredentials, the deployment might fail when trying to perform a login.

6.7. Obtaining container images from private registries

Some container image registries require authentication to access images. In this situation, use the ContainerImageRegistryCredentials parameter in your containers-prepare-parameter.yaml environment file.

parameter_defaults:
  ContainerImagePrepare:
  - (strategy one)
  - (strategy two)
  - (strategy three)
  ContainerImageRegistryCredentials:
    registry.example.com:
      username: "p@55w0rd!"
Important

Private registries require push_destination set to true for their respective strategy in the ContainerImagePrepare.

The ContainerImageRegistryCredentials parameter uses a set of keys based on the private registry URL. Each private registry URL uses its own key and value pair to define the username (key) and password (value). This provides a method to specify credentials for multiple private registries.

parameter_defaults:
  ...
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      myuser: 'p@55w0rd!'
    registry.internalsite.com:
      myuser2: '0th3rp@55w0rd!'
    '192.0.2.1:8787':
      myuser3: '@n0th3rp@55w0rd!'
Important

The default ContainerImagePrepare parameter pulls container images from registry.redhat.io, which requires authentication.

The ContainerImageRegistryLogin parameter is used to control whether the system needs to log in to the remote registry to fetch the containers.

parameter_defaults:
  ...
  ContainerImageRegistryLogin: true
Important

You must set ContainerImageRegistryLogin to true if push_destination is not configured for a given strategy. If push_destination is configured in a ContainerImagePrepare strategy and the ContainerImageRegistryCredentials parameter is configured, the system logs in to fetch the containers and pushes them to the remote system. If the overcloud nodes do not have network connectivity to the registry hosts defined in the ContainerImageRegistryCredentials, set push_destination to true and ContainerImageRegistryLogin to false .

6.8. Obtaining transitional containers for upgrades

The upgrade requires containers from previous versions of Red Hat OpenStack Platform and Red Hat Ceph Storage. These containers help transition to Red Hat OpenStack Platform 16.1.

Procedure

  1. Log in to your undercloud host as the stack user.
  2. Edit the containers-prepare-parameter.yaml file.
  3. Add the following parameters to the set in the ContainerImagePrepare parameter:

    parameter_defaults:
      ContainerImagePrepare:
      - push_destination: true
        set:
          ...
          name_prefix_stein: openstack-
          name_suffix_stein: ''
          namespace_stein: registry.redhat.io/rhosp15-rhel8
          tag_stein: 15.0
          ceph3_namespace: registry.redhat.io/rhceph
          ceph3_tag: latest
          ceph3_image: rhceph-3-rhel7
          ...
    • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
    • The ceph3_* parameters define the current Red Hat Ceph Storage container images that the overcloud uses. The overcloud requires these containers until the transition from Red Hat Ceph Storage 3 to 4.
    • If you use Red Hat Satellite for container image storage, set the namespaces to the image locations on the Red Hat Satellite server.
  4. Change the neutron_driver parameter to openvswitch:

    parameter_defaults:
      ContainerImagePrepare:
      - push_destination: true
        set:
          ...
          neutron_driver: openvswitch
          ...

    The upgrade retains Open vSwitch compatibility throughout the process. After you complete the upgrade to Red Hat OpenStack Platform 16.1, you migrate the overcloud from Open vSwitch to Open Virtual Network (OVN).

  5. Save the containers-prepare-parameter.yaml file.

6.9. Updating the undercloud.conf file

You can continue using the original undercloud.conf file from your Red Hat OpenStack Platform 13 environment, but you must modify the file to retain compatibility with Red Hat OpenStack Platform 16.1.

Procedure

  1. Log in to your undercloud host as the stack user.
  2. Edit the undercloud.conf file.
  3. Add the following parameter to the DEFAULT section in the file:

    container_images_file = /home/stack/containers-prepare-parameter.yaml

    This parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.

  4. Check the local_interface parameter if you have migrated to a predictable NIC naming convention.
  5. Check all other parameters in the file for any changes.
  6. Save the file.

6.10. Director configuration parameters

The following list contains information about parameters for configuring the undercloud.conf file. Keep all parameters within their relevant sections to avoid errors.

Defaults

The following parameters are defined in the [DEFAULT] section of the undercloud.conf file:

additional_architectures

A list of additional (kernel) architectures that an overcloud supports. Currently the overcloud supports ppc64le architecture.

Note

When you enable support for ppc64le, you must also set ipxe_enabled to False

certificate_generation_ca
The certmonger nickname of the CA that signs the requested certificate. Use this option only if you have set the generate_service_certificate parameter. If you select the local CA, certmonger extracts the local CA certificate to /etc/pki/ca-trust/source/anchors/cm-local-ca.pem and adds the certificate to the trust chain.
clean_nodes
Defines whether to wipe the hard drive between deployments and after introspection.
cleanup
Cleanup temporary files. Set this to False to leave the temporary files used during deployment in place after you run the deployment command. This is useful for debugging the generated files or if errors occur.
container_cli
The CLI tool for container management. Leave this parameter set to podman. Red Hat Enterprise Linux 8.2 only supports podman.
container_healthcheck_disabled
Disables containerized service health checks. Red Hat recommends that you enable health checks and leave this option set to false.
container_images_file

Heat environment file with container image information. This file can contain the following entries:

  • Parameters for all required container images
  • The ContainerImagePrepare parameter to drive the required image preparation. Usually the file that contains this parameter is named containers-prepare-parameter.yaml.
container_insecure_registries
A list of insecure registries for podman to use. Use this parameter if you want to pull images from another source, such as a private container registry. In most cases, podman has the certificates to pull container images from either the Red Hat Container Catalog or from your Satellite server if the undercloud is registered to Satellite.
container_registry_mirror
An optional registry-mirror configured that podman uses.
custom_env_files
Additional environment files that you want to add to the undercloud installation.
deployment_user
The user who installs the undercloud. Leave this parameter unset to use the current default user stack.
discovery_default_driver
Sets the default driver for automatically enrolled nodes. Requires the enable_node_discovery parameter to be enabled and you must include the driver in the enabled_hardware_types list.
enable_ironic; enable_ironic_inspector; enable_mistral; enable_nova; enable_tempest; enable_validations; enable_zaqar
Defines the core services that you want to enable for director. Leave these parameters set to true.
enable_node_discovery
Automatically enroll any unknown node that PXE-boots the introspection ramdisk. New nodes use the fake_pxe driver as a default but you can set discovery_default_driver to override. You can also use introspection rules to specify driver information for newly enrolled nodes.
enable_novajoin
Defines whether to install the novajoin metadata service in the undercloud.
enable_routed_networks
Defines whether to enable support for routed control plane networks.
enable_swift_encryption
Defines whether to enable Swift encryption at-rest.
enable_telemetry
Defines whether to install OpenStack Telemetry services (gnocchi, aodh, panko) in the undercloud. Set the enable_telemetry parameter to true if you want to install and configure telemetry services automatically. The default value is false, which disables telemetry on the undercloud. This parameter is required if you use other products that consume metrics data, such as Red Hat CloudForms.
enabled_hardware_types
A list of hardware types that you want to enable for the undercloud.
generate_service_certificate
Defines whether to generate an SSL/TLS certificate during the undercloud installation, which is used for the undercloud_service_certificate parameter. The undercloud installation saves the resulting certificate /etc/pki/tls/certs/undercloud-[undercloud_public_vip].pem. The CA defined in the certificate_generation_ca parameter signs this certificate.
heat_container_image
URL for the heat container image to use. Leave unset.
heat_native
Run host-based undercloud configuration using heat-all. Leave as true.
hieradata_override
Path to hieradata override file that configures Puppet hieradata on the director, providing custom configuration to services beyond the undercloud.conf parameters. If set, the undercloud installation copies this file to the /etc/puppet/hieradata directory and sets it as the first file in the hierarchy. For more information about using this feature, see Configuring hieradata on the undercloud.
inspection_extras
Defines whether to enable extra hardware collection during the inspection process. This parameter requires the python-hardware or python-hardware-detect packages on the introspection image.
inspection_interface
The bridge that director uses for node introspection. This is a custom bridge that the director configuration creates. The LOCAL_INTERFACE attaches to this bridge. Leave this as the default br-ctlplane.
inspection_runbench
Runs a set of benchmarks during node introspection. Set this parameter to true to enable the benchmarks. This option is necessary if you intend to perform benchmark analysis when inspecting the hardware of registered nodes.
ipa_otp
Defines the one-time password to register the undercloud node to an IPA server. This is required when enable_novajoin is enabled.
ipv6_address_mode

IPv6 address configuration mode for the undercloud provisioning network. The following list contains the possible values for this parameter:

  • dhcpv6-stateless - Address configuration using router advertisement (RA) and optional information using DHCPv6.
  • dhcpv6-stateful - Address configuration and optional information using DHCPv6.
ipxe_enabled
Defines whether to use iPXE or standard PXE. The default is true, which enables iPXE. Set this parameter to false to use standard PXE.
local_interface

The chosen interface for the director Provisioning NIC. This is also the device that director uses for DHCP and PXE boot services. Change this value to your chosen device. To see which device is connected, use the ip addr command. For example, this is the result of an ip addr command:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:75:24:09 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.178/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 3462sec preferred_lft 3462sec
    inet6 fe80::5054:ff:fe75:2409/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noop state DOWN
    link/ether 42:0b:c2:a5:c1:26 brd ff:ff:ff:ff:ff:ff

In this example, the External NIC uses eth0 and the Provisioning NIC uses eth1, which is currently not configured. In this case, set the local_interface to eth1. The configuration script attaches this interface to a custom bridge defined with the inspection_interface parameter.

local_ip
The IP address defined for the director Provisioning NIC. This is also the IP address that director uses for DHCP and PXE boot services. Leave this value as the default 192.168.24.1/24 unless you use a different subnet for the Provisioning network, for example, if this IP address conflicts with an existing IP address or subnet in your environment.
local_mtu
The maximum transmission unit (MTU) that you want to use for the local_interface. Do not exceed 1500 for the undercloud.
local_subnet
The local subnet that you want to use for PXE boot and DHCP interfaces. The local_ip address should reside in this subnet. The default is ctlplane-subnet.
net_config_override
Path to network configuration override template. If you set this parameter, the undercloud uses a JSON format template to configure the networking with os-net-config and ignores the network parameters set in undercloud.conf. Use this parameter when you want to configure bonding or add an option to the interface. See /usr/share/instack-undercloud/templates/net-config.json.template for an example.
networks_file
Networks file to override for heat.
output_dir
Directory to output state, processed heat templates, and Ansible deployment files.
overcloud_domain_name

The DNS domain name that you want to use when you deploy the overcloud.

Note

When you configure the overcloud, you must set the CloudDomain parameter to a matching value. Set this parameter in an environment file when you configure your overcloud.

roles_file
The roles file that you want to use to override the default roles file for undercloud installation. It is highly recommended to leave this parameter unset so that the director installation uses the default roles file.
scheduler_max_attempts
The maximum number of times that the scheduler attempts to deploy an instance. This value must be greater or equal to the number of bare metal nodes that you expect to deploy at once to avoid potential race conditions when scheduling.
service_principal
The Kerberos principal for the service using the certificate. Use this parameter only if your CA requires a Kerberos principal, such as in FreeIPA.
subnets
List of routed network subnets for provisioning and introspection. The default value includes only the ctlplane-subnet subnet. For more information, see Subnets.
templates
Heat templates file to override.
undercloud_admin_host
The IP address or hostname defined for director Admin API endpoints over SSL/TLS. The director configuration attaches the IP address to the director software bridge as a routed IP address, which uses the /32 netmask.
undercloud_debug
Sets the log level of undercloud services to DEBUG. Set this value to true to enable DEBUG log level.
undercloud_enable_selinux
Enable or disable SELinux during the deployment. It is highly recommended to leave this value set to true unless you are debugging an issue.
undercloud_hostname
Defines the fully qualified host name for the undercloud. If set, the undercloud installation configures all system host name settings. If left unset, the undercloud uses the current host name, but you must configure all system host name settings appropriately.
undercloud_log_file
The path to a log file to store the undercloud install and upgrade logs. By default, the log file is install-undercloud.log in the home directory. For example, /home/stack/install-undercloud.log.
undercloud_nameservers
A list of DNS nameservers to use for the undercloud hostname resolution.
undercloud_ntp_servers
A list of network time protocol servers to help synchronize the undercloud date and time.
undercloud_public_host
The IP address or hostname defined for director Public API endpoints over SSL/TLS. The director configuration attaches the IP address to the director software bridge as a routed IP address, which uses the /32 netmask.
undercloud_service_certificate
The location and filename of the certificate for OpenStack SSL/TLS communication. Ideally, you obtain this certificate from a trusted certificate authority. Otherwise, generate your own self-signed certificate.
undercloud_timezone
Host timezone for the undercloud. If you do not specify a timezone, director uses the existing timezone configuration.
undercloud_update_packages
Defines whether to update packages during the undercloud installation.

Subnets

Each provisioning subnet is a named section in the undercloud.conf file. For example, to create a subnet called ctlplane-subnet, use the following sample in your undercloud.conf file:

[ctlplane-subnet]
cidr = 192.168.24.0/24
dhcp_start = 192.168.24.5
dhcp_end = 192.168.24.24
inspection_iprange = 192.168.24.100,192.168.24.120
gateway = 192.168.24.1
masquerade = true

You can specify as many provisioning networks as necessary to suit your environment.

cidr
The network that director uses to manage overcloud instances. This is the Provisioning network, which the undercloud neutron service manages. Leave this as the default 192.168.24.0/24 unless you use a different subnet for the Provisioning network.
masquerade

Defines whether to masquerade the network defined in the cidr for external access. This provides the Provisioning network with a degree of network address translation (NAT) so that the Provisioning network has external access through director.

Note

The director configuration also enables IP forwarding automatically using the relevant sysctl kernel parameter.

dhcp_start; dhcp_end
The start and end of the DHCP allocation range for overcloud nodes. Ensure that this range contains enough IP addresses to allocate your nodes.
dhcp_exclude
IP addresses to exclude in the DHCP allocation range.
dns_nameservers
DNS nameservers specific to the subnet. If no nameservers are defined for the subnet, the subnet uses nameservers defined in the undercloud_nameservers parameter.
gateway
The gateway for the overcloud instances. This is the undercloud host, which forwards traffic to the External network. Leave this as the default 192.168.24.1 unless you use a different IP address for director or want to use an external gateway directly.
host_routes
Host routes for the Neutron-managed subnet for the overcloud instances on this network. This also configures the host routes for the local_subnet on the undercloud.
inspection_iprange
Temporary IP range for nodes on this network to use during the inspection process. This range must not overlap with the range defined by dhcp_start and dhcp_end but must be in the same IP subnet.

6.11. Running the director upgrade

Complete the following steps to upgrade the director.

Procedure

  1. Run the following command to upgrade the director on the undercloud:

    $ openstack undercloud upgrade

    This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

    Note

    The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the -y option:

    $ openstack undercloud upgrade -y
  2. The script also starts all OpenStack Platform service containers on the undercloud automatically. You manage each service through a systemd resource. Check the systemd resources:

    $ sudo systemctl list-units "tripleo_*"

    Each systemd service controls a container. Check the enabled containers using the following command:

    $ sudo podman ps
  3. The script adds the stack user to the docker group to ensure that the stack user has access to container management commands. Refresh the stack user permissions with the following command:

    $ exec su -l stack

    The command prompts you to log in again. Enter the stack user password.

  4. To initialize the stack user to use the command line tools, run the following command:

    $ source ~/stackrc

    The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

    (undercloud) $

The director upgrade is complete.

Part II. Preparing for overcloud upgrade

Chapter 7. Initial steps for overcloud preparation

You must complete some initial steps to prepare for the overcloud upgrade.

7.1. Preparing for overcloud service downtime

The overcloud upgrade process disables the main services at key points. This means you cannot use any overcloud services to create new resources during the upgrade duration. Workloads running in the overcloud remain active during this period, which means instances continue to run through the upgrade duration.

It is important to plan a maintenance window to ensure no users can access the overcloud services during the upgrade duration.

Affected by overcloud upgrade

  • OpenStack Platform services

Unaffected by overcloud upgrade

  • Instances running during the upgrade
  • Ceph Storage OSDs (backend storage for instances)
  • Linux networking
  • Open vSwitch networking
  • Undercloud

7.2. Selecting Compute nodes for upgrade testing

The overcloud upgrade process allows you to either:

  • Upgrade all nodes in a role
  • Individual nodes separately

To ensure a smooth overcloud upgrade process, it is useful to test the upgrade on a few individual Compute nodes in your environment before upgrading all Compute nodes. This ensures no major issues occur during the upgrade while maintaining minimal downtime to your workloads.

Use the following recommendations to help choose test nodes for the upgrade:

  • Select two or three Compute nodes for upgrade testing
  • Select nodes without any critical instances running
  • If necessary, migrate critical instances from the selected test Compute nodes to other Compute nodes

7.3. Validating the pre-upgrade requirements

Run the pre-upgrade validation group to check the pre-upgrade requirements.

Procedure

  1. Source the stackrc file.

    $ source ~/stackrc
  2. Run the openstack tripleo validator run command with the --group pre-upgrade option:

    $ openstack tripleo validator run --group pre-upgrade
  3. Review the results of the validation report.
Important

A FAILED validation does not prevent you from deploying or running Red Hat OpenStack Platform. However, a FAILED validation can indicate a potential issue with a production environment.

7.4. Creating an overcloud inventory file

Generate an Ansible inventory file of all nodes in your environment with the tripleo-ansible-inventory command.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file.

    $ source ~/stackrc
  3. Create a static inventory file of all nodes:

    $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml
  4. To execute Ansible playbooks on your environment, run the ansible-playbook command and include the full path of the dynamic inventory tool using the -i option. For example:

    (undercloud) $ ansible-playbook -i ~/inventory.yaml PLAYBOOK

Chapter 8. Configuring the overcloud for a Leapp upgrade

You must complete some Leapp configuration to prepare for the overcloud upgrade.

8.1. Creating an upgrades environment file

The upgrade process uses an environment file to enable the upgrade process and configure specific upgrade parameters.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create an environment file called upgrades-environment.yaml in your templates directory:

    $ touch templates/upgrades-environment.yaml
  3. Edit the file and add the following mandatory content:

    parameter_defaults:
      UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
      CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
    • UpgradeLeappCommandOptions sets additional options to pass to the Leapp tool. In this context, set this parameter to enable Red Hat OpenStack Platform repositories to help with the Leapp transition.
    • CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.
  4. Save the upgrades-environment.yaml file.

8.2. Upgrade parameters

ParameterDescription

UpgradeInitCommand

Command or script snippet to run on all overcloud nodes to initialize the upgrade process. For example, a repository switch.

UpgradeInitCommonCommand

Common commands required by the upgrades process. This should not normally be modified by the operator and is set and unset in the major-upgrade-composable-steps.yaml and major-upgrade-converge.yaml environment files.

UpgradeLeappCommandOptions

Additional command line options to append to the Leapp command.

UpgradeLeappDebug

Print debugging output when running Leapp. The default value is True.

UpgradeLeappDevelSkip

Skip Leapp checks by setting env variables when running Leapp in development/testing. For example, LEAPP_DEVEL_SKIP_RHSM=1.

UpgradeLeappEnabled

Use Leapp for operating system upgrade. The default value is False.

UpgradeLeappRebootTimeout

Timeout (seconds) for the OS upgrade phase via Leapp. The default value is 3600.

UpgradeLeappToInstall

List of packages to install after Leapp upgrade.

UpgradeLeappToRemove

List of packages to remove during Leapp upgrade.

8.3. Copying the Leapp data to the overcloud nodes

Each overcloud node requires the Leapp data files. Copy the leapp-data files from the undercloud to each overcloud.

Prerequisites

  • An Ansible inventory file for your overcloud nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create a file called playbook-leapp-data.yaml and paste the following content in the file:

    ---
    - name: Copy Leapp data files to overcloud
      hosts: overcloud
      become: yes
      tasks:
        - name: Create the Leapp files directory
          file:
            path: /etc/leapp/files
            state: directory
        - name: Copy pes-events.json
          copy:
            src: /etc/leapp/files/pes-events.json
            dest: /etc/leapp/files/pes-events.json
        - name: Copy
          copy:
            src: /etc/leapp/files/repomap.csv
            dest: /etc/leapp/files/repomap.csv
  3. Run the playbook:

    $ ansible-playbook -i ~/inventory.yaml playbook-leapp-data.yaml

8.4. Using predictable NIC names for overcloud nodes

Before you run the Leapp upgrade on overcloud nodes, you must check for kernel-based NIC names, which usually contain a eth prefix. These NIC names are usually unpredictable in terms of NIC assignments. The Leapp upgrade fails if any NICs on the node use the eth prefix.

Prerequisites

  • The playbook-nics.yaml playbook created during the undercloud preparation process.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Run the playbook-nics.yaml playbook on all overcloud nodes:

    $ ansible-playbook -i ~/inventory.yaml playbook-nics.yaml

    The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

    $ ansible-playbook -i ~/inventory.yaml -e prefix="mynic" playbook-nics.yaml
  3. Reboot overcloud nodes using the standard reboot procedures. For more information, see "Rebooting nodes".

8.5. Setting the SSH root permission parameter on the overcloud

The Leapp upgrade checks whether the PermitRootLogin parameter exists in the /etc/ssh/sshd_config file on each node. You must explicitly set this parameter to either yes or no.

For security purposes, set this parameter to no to disable SSH access to the root user on overcloud nodes.

Prerequisites

  • An Ansible inventory file for your overcloud nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create a file called playbook-ssh.yaml and paste the following content in the file:

    ---
    - name: Configure SSH PermitRootLogin parameter
      hosts: overcloud
      become: yes
      tasks:
        - name: Set the PermitRootLogin parameter to no
          lineinfile:
            path: /etc/ssh/sshd_config
            state: present
            line: "PermitRootLogin no"
  3. Run the playbook:

    $ ansible-playbook -i ~/inventory.yaml playbook-ssh.yaml

8.6. Configuring access to the undercloud registry

Add the control plane host name of the undercloud to the DockerInsecureRegistryAddress parameter. Place this parameter in the containers-prepare-parameter.yaml file to ensure that the parameter is included in future overcloud deployments.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Obtain the control plane host name on the undercloud:

    $ sudo hiera container_image_prepare_node_names
    ["undercloud.ctlplane.localdomain"]
  3. Edit the containers-prepare-parameter.yaml file and add the DockerInsecureRegistryAddress parameter with a YAML list that contains the undercloud control plane host name:

    parameter_defaults:
      DockerInsecureRegistryAddress:
      - undercloud.ctlplane.localdomain:8787
      ...

    You must also append the port number of the overcloud registry to the host name value. The port number is 8787.

Chapter 9. Updating composable services

You must complete some composable service configuration to prepare for the overcloud upgrade.

9.1. Updating composable services in custom roles_data files

This section contains information about new and deprecated composable services.

  • If you use the default roles_data file, these services are included automatically.
  • If you use a custom roles_data file, add the new services and remove the deprecated services for each relevant role.

Controller Nodes

The following services have been deprecated for Controller nodes. Remove them from your Controller role.

ServiceReason

OS::TripleO::Services::AodhApi

OS::TripleO::Services::AodhEvaluator

OS::TripleO::Services::AodhListener

OS::TripleO::Services::AodhNotifier

OS::TripleO::Services::CeilometerApi

OS::TripleO::Services::CeilometerCollector

OS::TripleO::Services::CeilometerExpirer

OS::TripleO::Services::MongoDb

OS::TripleO::Services::PankoApi

OpenStack Platform no longer includes OpenStack Telemetry services and now uses Service Telemetry Framework (STF) for metrics and monitoring.

OS::TripleO::Services::Congress

Congress is no longer supported.

OS::TripleO::Services::GlanceRegistry

This service is no longer supported due to OpenStack Platform Image Service (glance) API v2.

OS::TripleO::Services::NeutronCorePluginPlumgrid

OS::TripleO::Services::NeutronCorePluginMidonet

Deprecated plug-ins for OpenStack Networking (neutron).

OS::TripleO::Services::NeutronLbaasv2Agent

OS::TripleO::Services::NeutronLbaasv2Api

OpenStack Networking (neutron) Load Balancing as a Service deprecated in favour of Octavia.

OS::TripleO::Services::NovaConsoleauth

This service is now deprecated.

OS::TripleO::Services::NovaPlacement

Deprecated in favor of OS::TripleO::Services::PlacementApi.

OS::TripleO::Services::OpenDaylightApi

OS::TripleO::Services::OpenDaylightOvs

OpenDaylight is no longer supported.

OS::TripleO::Services::RabbitMQ

This service has been substituted for two new services:

OS::TripleO::Services::OsloMessagingRpc

OS::TripleO::Services::OsloMessagingNotify

OS::TripleO::Services::SkydiveAgent

OS::TripleO::Services::SkydiveAnalyzer

Skydive is no longer supported.

OS::TripleO::Services::Tacker

Tacker is no longer supported.

The following services are new for Controller nodes. Add them to your Controller role.

ServiceReason

OS::TripleO::Services::CephGrafana

Tasks to enable Ceph Dashboard service.

OS::TripleO::Services::CinderBackendDellEMCPowermax

OS::TripleO::Services::CinderBackendDellEMCSc

OS::TripleO::Services::CinderBackendNVMeOF

New backends for Block Storage (cinder).

OS::TripleO::Services::ContainerImagePrepare

Run the commands to automatically pull and prepare container images relevant to the services in your overcloud.

OS::TripleO::Services::DesignateApi

OS::TripleO::Services::DesignateCentral

OS::TripleO::Services::DesignateProducer

OS::TripleO::Services::DesignateWorker

OS::TripleO::Services::DesignateMDNS

OS::TripleO::Services::DesignateSink

Services for DNS-as-a-Service (designate).

OS::TripleO::Services::IronicInspector

Service for Bare Metal Introspection for the overcloud.

OS::TripleO::Services::IronicNeutronAgent

The networking agent for OpenStack Bare Metal (ironic).

OS::TripleO::Services::NeutronAgentsIBConfig

Service for OpenStack Networking (neutron) agent for Mellanox InfiniBand.

OS::TripleO::Services::OpenStackClients

Service for installing the Red Hat OpenStack Platform command line tools.

OS::TripleO::Services::OsloMessagingRpc

OS::TripleO::Services::OsloMessagingNotify

Replacement services for the OS::TripleO::Services::RabbitMQ service.

OS::TripleO::Services::PlacementApi

Service for the Placement API.

Compute Nodes

The following services have been deprecated for Compute nodes. Remove them from your Compute role.

ServiceReason

OS::TripleO::Services::OpenDaylightOvs

OpenDaylight is no longer supported.

OS::TripleO::Services::SkydiveAgent

Skydive is no longer supported.

The following services are new for Compute nodes. Add them to your Compute role.

ServiceReason

OS::TripleO::Services::NovaAZConfig

Service to configure host aggregate and availability zone in OpenStack Compute (nova).

All Nodes

The following services have been deprecated for all nodes. Remove them from all roles.

ServiceReason

OS::TripleO::Services::Docker

Replaced with Podman.

OS::TripleO::Services::Fluentd

Deprecated in favour of OS::TripleO::Services::Rsyslog.

OS::TripleO::Services::Ntp

Deprecated in favour of OS::TripleO::Services::Timesync.

OS::TripleO::Services::SensuClient

Deprecated service.

OS::TripleO::Services::Ptp

Deprecated in favour of OS::TripleO::Services::Timesync.

The following services are new for all nodes. Add them to all roles.

ServiceReason

OS::TripleO::Services::BootParams

Service to set Kernel arguments, Tuned profiles, and CPU isolation.

OS::TripleO::Services::Collectd

Service to configure Collectd.

OS::TripleO::Services::Multipathd

Provides a Multipathd service, which is disabled by default

OS::TripleO::Services::Podman

Service to install and enable Podman.

OS::TripleO::Services::Rear

Service to install and enable Relax-and-Recover (ReaR) backup and restore tools.

OS::TripleO::Services::Rsyslog

Service to configure centralized log collection.

OS::TripleO::Services::Timesync

Service to enable time synchronization method, which is Chronyd by default.

9.2. Updating composable services in custom environment files

If you have any custom environment files with resource_registry sections, check the resource_registry sections for changes in composable service template mappings. The composable service files for Red Hat OpenStack Platform 16.1 reside in a new location within /usr/share/openstack-tripleo-heat-templates/:

Red Hat OpenStack Platform 13Red Hat OpenStack Platform 16.1

docker/services/

deployment

The deployment directory contains a set of sub-directories to group composable services. For example, the keystone sub-directory contains composable service templates for OpenStack Identity (keystone).

To remap composable services in custom environment files, check the template location for the current service mapping and edit the mapping to the new location. This procedure uses ceph-mgr.yaml as an example.

Important

This procedure acts as guidance only for remapping composable services. If you are unsure of any mapping, contact Red Hat and request advice on the correct mapping.

Procedure

  1. Search for custom environment files that use composable services. You usually store custom environment files in the /home/stack/templates directory:

    $ cd ~/templates/
    $ grep "OS::TripleO::Services" *

    In this scenario, one of the files shows an outdated mapping:

      OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mgr.yaml
  2. Identify the new ceph-mgr.yaml location in /usr/share/openstack-tripleo-heat-templates/. This file is now located in the `deployment/ceph-ansible' directory:

    $ find /usr/share/openstack-tripleo-heat-templates/ -name ceph-mgr.yaml
    /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml
  3. Edit the service in the custom environment file:

    resource_registry:
      OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

    Save the file.

Chapter 10. Updating overcloud registration to the Red Hat Customer Portal

Red Hat OpenStack Platform 16.1 uses Ansible-based methods to register overcloud nodes to the Red Hat Customer Portal.

10.1. Red Hat Subscription Manager (RHSM) composable service

The rhsm composable service provides a method to register overcloud nodes through Ansible. Each role in the default roles_data file contains a OS::TripleO::Services::Rhsm resource, which is disabled by default. To enable the service, register the resource to the rhsm composable service file:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml

The rhsm composable service accepts a RhsmVars parameter, which allows you to define multiple sub-parameters relevant to your registration. For example:

parameter_defaults:
  RhsmVars:
    rhsm_repos:
      - rhel-8-for-x86_64-baseos-eus-rpms
      - rhel-8-for-x86_64-appstream-eus-rpms
      - rhel-8-for-x86_64-highavailability-eus-rpms
      - ansible-2.9-for-rhel-8-x86_64-rpms
      - advanced-virt-for-rhel-8-x86_64-rpms
      - openstack-16.1-for-rhel-8-x86_64-rpms
      - rhceph-4-osd-for-rhel-8-x86_64-rpms
      - rhceph-4-mon-for-rhel-8-x86_64-rpms
      - rhceph-4-tools-for-rhel-8-x86_64-rpms
      - fast-datapath-for-rhel-8-x86_64-rpms
    rhsm_username: "myusername"
    rhsm_password: "p@55w0rd!"
    rhsm_org_id: "1234567"
    rhsm_release: 8.2

You can also use the RhsmVars parameter in combination with role-specific parameters (e.g. ControllerParameters) to provide flexibility when enabling specific repositories for different nodes types.

10.2. RhsmVars sub-parameters

See the role documentation to learn about all Ansible parameters.

rhsmDescription

rhsm_method

Choose the registration method. Either portal, satellite, or disable.

rhsm_org_id

The organization to use for registration. To locate this ID, run sudo subscription-manager orgs from the undercloud node. Enter your Red Hat credentials when prompted, and use the resulting Key value.

rhsm_pool_ids

The subscription pool ID to use. Use this if not auto-attaching subscriptions. To locate this ID, run sudo subscription-manager list --available --all --matches="OpenStack" from the undercloud node, and use the resulting Pool ID value.

rhsm_activation_key

The activation key to use for registration. Does not work when rhsm_repos is configured.

rhsm_autosubscribe

Automatically attach compatible subscriptions to this system. Set to true to enable.

rhsm_baseurl

The base URL for obtaining content. The default is the Red Hat Content Delivery Network URL. If using a Satellite server, change this value to the base URL of your Satellite server content repositories.

rhsm_server_hostname

The hostname of the subscription management service for registration. The default is the Red Hat Subscription Management hostname. If using a Satellite server, change this value to your Satellite server hostname.

rhsm_repos

A list of repositories to enable. Does not work when rhsm_activation_key is configured.

rhsm_username

The username for registration. If possible, use activation keys for registration.

rhsm_password

The password for registration. If possible, use activation keys for registration.

rhsm_release

Red Hat Enterprise Linux release for pinning the repositories. This is set to 8.2 for Red Hat OpenStack Platform 16.1.

rhsm_rhsm_proxy_hostname

The hostname for the HTTP proxy. For example: proxy.example.com.

rhsm_rhsm_proxy_port

The port for HTTP proxy communication. For example: 8080.

rhsm_rhsm_proxy_user

The username to access the HTTP proxy.

rhsm_rhsm_proxy_password

The password to access the HTTP proxy.

10.3. Switching to the rhsm composable service

The previous rhel-registration method runs a bash script to handle the overcloud registration. The scripts and environment files for this method are located in the core Heat template collection at /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/.

Complete the following steps to switch from the rhel-registration method to the rhsm composable service.

Procedure

  1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

    • rhel-registration/environment-rhel-registration.yaml
    • rhel-registration/rhel-registration-resource-registry.yaml
  2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

    - name: Controller
      description: |
        Controller role that has all the controller services loaded and handles
        Database, Messaging and Network functions.
      CountDefault: 1
      ...
      ServicesDefault:
        ...
        - OS::TripleO::Services::Rhsm
        ...
  3. Add the environment file for rhsm composable service parameters to future deployment operations.

This method replaces the rhel-registration parameters with the rhsm service parameters and changes the Heat resource that enables the service from:

resource_registry:
  OS::TripleO::NodeExtraConfig: rhel-registration.yaml

To:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml

You can also include the /usr/share/openstack-tripleo-heat-templates/environments/rhsm.yaml environment file with your deployment to enable the service.

10.4. rhel-registration to rhsm mappings

rhel-registrationrhsm / RhsmVars

rhel_reg_method

rhsm_method

rhel_reg_org

rhsm_org_id

rhel_reg_pool_id

rhsm_pool_ids

rhel_reg_activation_key

rhsm_activation_key

rhel_reg_auto_attach

rhsm_autosubscribe

rhel_reg_sat_url

rhsm_satellite_url

rhel_reg_repos

rhsm_repos

rhel_reg_user

rhsm_username

rhel_reg_password

rhsm_password

rhel_reg_release

rhsm_release

rhel_reg_http_proxy_host

rhsm_rhsm_proxy_hostname

rhel_reg_http_proxy_port

rhsm_rhsm_proxy_port

rhel_reg_http_proxy_username

rhsm_rhsm_proxy_user

rhel_reg_http_proxy_password

rhsm_rhsm_proxy_password

10.5. Registering the overcloud with the rhsm composable service

Use the following procedure to create an environment file that enables and configures the rhsm composable service. The director uses this environment file to register and subscribe your nodes.

Procedure

  1. Create an environment file (templates/rhsm.yml) to store the configuration.
  2. Include your configuration in the environment file. For example:

    resource_registry:
      OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
    parameter_defaults:
      RhsmVars:
        rhsm_repos:
          - rhel-8-for-x86_64-baseos-eus-rpms
          - rhel-8-for-x86_64-appstream-eus-rpms
          - rhel-8-for-x86_64-highavailability-eus-rpms
          - ansible-2.9-for-rhel-8-x86_64-rpms
          - advanced-virt-for-rhel-8-x86_64-rpms
          - openstack-16.1-for-rhel-8-x86_64-rpms
          - rhceph-4-osd-for-rhel-8-x86_64-rpms
          - rhceph-4-mon-for-rhel-8-x86_64-rpms
          - rhceph-4-tools-for-rhel-8-x86_64-rpms
          - fast-datapath-for-rhel-8-x86_64-rpms
        rhsm_username: "myusername"
        rhsm_password: "p@55w0rd!"
        rhsm_org_id: "1234567"
        rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
        rhsm_method: "portal"
        rhsm_release: 8.2

    The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

    The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

  3. Save the environment file.

Chapter 11. Updating overcloud registration to Red Hat Satellite

Red Hat OpenStack Platform 16.1 uses Ansible-based methods to register overcloud nodes to a Red Hat Satellite 6 server.

11.1. Red Hat Subscription Manager (RHSM) composable service

The rhsm composable service provides a method to register overcloud nodes through Ansible. Each role in the default roles_data file contains a OS::TripleO::Services::Rhsm resource, which is disabled by default. To enable the service, register the resource to the rhsm composable service file:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml

The rhsm composable service accepts a RhsmVars parameter, which allows you to define multiple sub-parameters relevant to your registration. For example:

parameter_defaults:
  RhsmVars:
    rhsm_repos:
      - rhel-8-for-x86_64-baseos-eus-rpms
      - rhel-8-for-x86_64-appstream-eus-rpms
      - rhel-8-for-x86_64-highavailability-eus-rpms
      - ansible-2.9-for-rhel-8-x86_64-rpms
      - advanced-virt-for-rhel-8-x86_64-rpms
      - openstack-16.1-for-rhel-8-x86_64-rpms
      - rhceph-4-osd-for-rhel-8-x86_64-rpms
      - rhceph-4-mon-for-rhel-8-x86_64-rpms
      - rhceph-4-tools-for-rhel-8-x86_64-rpms
      - fast-datapath-for-rhel-8-x86_64-rpms
    rhsm_username: "myusername"
    rhsm_password: "p@55w0rd!"
    rhsm_org_id: "1234567"
    rhsm_release: 8.2

You can also use the RhsmVars parameter in combination with role-specific parameters (e.g. ControllerParameters) to provide flexibility when enabling specific repositories for different nodes types.

11.2. RhsmVars sub-parameters

See the role documentation to learn about all Ansible parameters.

rhsmDescription

rhsm_method

Choose the registration method. Either portal, satellite, or disable.

rhsm_org_id

The organization to use for registration. To locate this ID, run sudo subscription-manager orgs from the undercloud node. Enter your Red Hat credentials when prompted, and use the resulting Key value.

rhsm_pool_ids

The subscription pool ID to use. Use this if not auto-attaching subscriptions. To locate this ID, run sudo subscription-manager list --available --all --matches="OpenStack" from the undercloud node, and use the resulting Pool ID value.

rhsm_activation_key

The activation key to use for registration. Does not work when rhsm_repos is configured.

rhsm_autosubscribe

Automatically attach compatible subscriptions to this system. Set to true to enable.

rhsm_baseurl

The base URL for obtaining content. The default is the Red Hat Content Delivery Network URL. If using a Satellite server, change this value to the base URL of your Satellite server content repositories.

rhsm_server_hostname

The hostname of the subscription management service for registration. The default is the Red Hat Subscription Management hostname. If using a Satellite server, change this value to your Satellite server hostname.

rhsm_repos

A list of repositories to enable. Does not work when rhsm_activation_key is configured.

rhsm_username

The username for registration. If possible, use activation keys for registration.

rhsm_password

The password for registration. If possible, use activation keys for registration.

rhsm_release

Red Hat Enterprise Linux release for pinning the repositories. This is set to 8.2 for Red Hat OpenStack Platform 16.1.

rhsm_rhsm_proxy_hostname

The hostname for the HTTP proxy. For example: proxy.example.com.

rhsm_rhsm_proxy_port

The port for HTTP proxy communication. For example: 8080.

rhsm_rhsm_proxy_user

The username to access the HTTP proxy.

rhsm_rhsm_proxy_password

The password to access the HTTP proxy.

11.3. Switching to the rhsm composable service

The previous rhel-registration method runs a bash script to handle the overcloud registration. The scripts and environment files for this method are located in the core Heat template collection at /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/.

Complete the following steps to switch from the rhel-registration method to the rhsm composable service.

Procedure

  1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

    • rhel-registration/environment-rhel-registration.yaml
    • rhel-registration/rhel-registration-resource-registry.yaml
  2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

    - name: Controller
      description: |
        Controller role that has all the controller services loaded and handles
        Database, Messaging and Network functions.
      CountDefault: 1
      ...
      ServicesDefault:
        ...
        - OS::TripleO::Services::Rhsm
        ...
  3. Add the environment file for rhsm composable service parameters to future deployment operations.

This method replaces the rhel-registration parameters with the rhsm service parameters and changes the Heat resource that enables the service from:

resource_registry:
  OS::TripleO::NodeExtraConfig: rhel-registration.yaml

To:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml

You can also include the /usr/share/openstack-tripleo-heat-templates/environments/rhsm.yaml environment file with your deployment to enable the service.

11.4. rhel-registration to rhsm mappings

rhel-registrationrhsm / RhsmVars

rhel_reg_method

rhsm_method

rhel_reg_org

rhsm_org_id

rhel_reg_pool_id

rhsm_pool_ids

rhel_reg_activation_key

rhsm_activation_key

rhel_reg_auto_attach

rhsm_autosubscribe

rhel_reg_sat_url

rhsm_satellite_url

rhel_reg_repos

rhsm_repos

rhel_reg_user

rhsm_username

rhel_reg_password

rhsm_password

rhel_reg_release

rhsm_release

rhel_reg_http_proxy_host

rhsm_rhsm_proxy_hostname

rhel_reg_http_proxy_port

rhsm_rhsm_proxy_port

rhel_reg_http_proxy_username

rhsm_rhsm_proxy_user

rhel_reg_http_proxy_password

rhsm_rhsm_proxy_password

11.5. Registering the overcloud to Red Hat Satellite

Use the following procedure to create an environment file that enables and configures the rhsm composable service to register nodes to Red Hat Satellite instead of the Red Hat Customer Portal.

Procedure

  1. Create an environment file (templates/rhsm.yml) to store the configuration.
  2. Include your configuration in the environment file. For example:

    resource_registry:
      OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
    parameter_defaults:
      RhsmVars:
        rhsm_activation_key: "myactivationkey"
        rhsm_method: "satellite"
        rhsm_org_id: "ACME"
        rhsm_server_hostname: satellite.example.com"
        rhsm_baseurl: "https://satellite.example.com/pulp/repos"
        rhsm_release: 8.2

    The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

    The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

  3. Save the environment file.

11.6. Preparing Leapp to use Satellite server

If you use Satellite 6 to host RPM content, complete these preparation steps to ensure a successful Leapp upgrade with Satellite.

Prerequisites

  • A Satellite server activation key linked to repositories for Red Hat OpenStack Platform 16.1 and Red Hat Enterprise Linux 8.2.
  • An Ansible inventory file for your overcloud nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Create a file called playbook-satellite.yaml and paste the following content in the file:

    - name: Pre-install leapp
      hosts: overcloud
      become: yes
      tasks:
        - name: Pre-install leapp
          yum:
            name:
              - leapp
              - leapp-repository
            state: installed
        - name: Remove katello-host-tools-fact-plugin
          yum:
            name:
              - katello-host-tools-fact-plugin
            state: removed
        - name: Register system
          import_role:
            name: redhat-subscription
          vars:
            rhsm_activation_key: "osp16-key"
            rhsm_method: "satellite"
            rhsm_org_id: "ACME"
            rhsm_server_hostname: "satellite.example.com"
            rhsm_baseurl: "https://satellite.example.com/pulp/repos"
            rhsm_release: "8.2"
            rhsm_force_register: True

    Modify the rhsm_* variables to suit your Satellite server.

  3. Run the playbook:

    $ ansible-playbook -i ~/inventory.yaml playbook-satellite.yaml

Chapter 12. Updating network configuration

You must complete some network configuration to prepare for the overcloud upgrade.

12.1. Updating network interface templates

Red Hat OpenStack Platform includes a script to automatically add the missing parameters to your NIC template files.

Procedure

  1. Run the following script on the undercloud:

    STACK_NAME="overcloud"
    ROLES_DATA="/usr/share/openstack-tripleo-heat-templates/roles_data.yaml"
    NIC_CONFIG_LINES=$(openstack stack environment show $STACK_NAME | grep "::Net::SoftwareConfig" | sed -E 's/ *OS::TripleO::// ; s/::Net::SoftwareConfig:// ; s/ http.*user-files/ /')
    echo "$NIC_CONFIG_LINES"
    echo "$NIC_CONFIG_LINES" | while read LINE; do
        ROLE=$(echo "$LINE" | awk '{print $1;}')
        NIC_CONFIG=$(echo "$LINE" | awk '{print $2;}')
    
        python3 /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
                --tht-dir /usr/share/openstack-tripleo-heat-templates \
                --roles-data $ROLES_DATA \
                --role-name "$ROLE" \
                --discard-comments yes \
                --template "$NIC_CONFIG"
    done
    • If you use a custom overcloud name, Set the STACK_NAME variable to the name of your overcloud. The default name for an overcloud stack is overcloud.
    • If you use a custom roles_data file, set the ROLES_DATA variable to the location of the custom file. If you use the default roles_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
    • Run /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py -h to see a list of options to add to the script.

12.2. Maintaining Open vSwitch compatibility during the upgrade

Red Hat OpenStack Platform 13 uses Open vSwitch (OVS) as the default ML2 back end for OpenStack Networking (neutron). Newer versions of Red Hat OpenStack Platform use Open Virtual Network (OVN), which expands upon OVS capabilities. However, to ensure a stable upgrade, you must maintain OVS functionality during the duration of the upgrade and then migrate to OVN after you complete the upgrade.

To maintain OVS compatibility during the upgrade, include the following environment file as part of your environment file collection:

  • /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml

Treat this file as part of your deployment until you have completed the migration to OVN. Include the file with all overcloud upgrade and deployment commands:

  • openstack overcloud upgrade prepare
  • openstack overcloud upgrade converge
  • openstack overcloud deploy
  • openstack overcloud update prepare
  • openstack overcloud update converge
  • Any other command that uses environment files.

Chapter 13. Temporary workaround configuration

Use the following procedures to implement temporary workarounds for known issues. These procedures use an environment file named workarounds.yaml, which you create in your templates directory:

$ touch templates/workarounds.yaml

13.1. Unsetting the Compute upgrade level

Unset the UpgradeLevelNovaCompute heat parameter to ensure successful live migration of workloads during the upgrade.

Important

This procedure is a temporary work around for BZ#1849235 - Unset UpgradeLevelNovaCompute or live migrations fail

Procedure

  1. Log in to the undercloud as the stack user.
  2. Edit the workarounds.yaml file and add the UpgradeLevelNovaCompute parameter to the parameter_defaults section:

    parameter_defaults:
      UpgradeLevelNovaCompute: ''
  3. Save the workarounds.yaml file.

Chapter 14. Final review before upgrade

Complete a final check of all preparation steps before you begin the upgrade.

14.1. New environment files to include with your deployment

In addition to your regular overcloud environment files, you must include new environment files to facilitate the upgrade to Red Hat OpenStack Platform 16.1

FileNotes

/home/stack/templates/upgrades-environment.yaml

This file contains the parameters specific to the upgrade. This file is necessary only for the duration of the upgrade. Discard this file after you run openstack overcloud upgrade converge.

/home/stack/templates/rhsm.yaml

This file contains registration and subscription information for the overcloud. This file registers your systems either to the Red Hat Customer Portal or a Red Hat Satellite server.

/home/stack/containers-prepare-parameter.yaml

The file that contains the source and preparation steps. This is the same file that you use with the undercloud upgrade.

/usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml

OpenStack Platform 16.1 uses Open Virtual Network (OVN) as the default networking backend. However, OpenStack Platform 13 used Open vSwitch (OVS). Include this file to retain OVS compatibility during the upgrade. The OpenStack Platform 16.1 documentation includes instructions on migrating from OVS to OVN after the upgrade.

/home/stack/templates/workarounds.yaml

Any temporary workarounds required during the overcloud upgrade.

Add these files to the end of your environment file listing when you run the following commands:

  • openstack overcloud upgrade prepare
  • openstack overcloud upgrade converge
  • openstack overcloud deploy

14.2. Environment files to remove from your deployment

Remove any environment files specific to your OpenStack Platform Red Hat OpenStack Platform 13:

  • Red Hat OpenStack Platform 13 container image list
  • Red Hat OpenStack Platform 13 Customer Portal or Satellite rhel-registration scripts

Remove these files from the list of environment files you include when you run the following commands:

  • openstack overcloud upgrade prepare
  • openstack overcloud upgrade converge
  • openstack overcloud deploy

14.3. Upgrade checklist

Use the following checklist to determine your readiness to upgrade the overcloud:

ItemComplete

Validated a working overcloud.

Y / N

Performed a Relax-and-Recover (ReaR) backup of the overcloud control plane.

Y / N

Implemented all preparation for Leapp, including:

  • Leapp environment file
  • Leapp data copied to the overcloud nodes
  • All overcloud nodes are now using predictable NIC names.
  • All overcloud nodes have PermitRootLogin set

Y / N

Updated your registration details to Red Hat OpenStack Platform 16.1 repositories and covnerted your environment file to use the Ansible-based method.

Y / N

Updated your network configuration templates.

Y / N

Updated your environment file list with new environment files for Red Hat OpenStack Platform 16.1.

Y / N

Removed old environment files only relevant to Red Hat OpenStack Platform 13, such as old Red Hat registration and container image location files.

Y / N

Part III. Upgrading the overcloud

Chapter 15. Upgrade command overview

The upgrade process involves different commands that you run at certain stages of process.

Important

This section only contains information about each command. You must run these commands in a specific order and provide options specific to your overcloud. Wait until you receive instructions to run these commands at the appropriate step.

15.1. openstack overcloud upgrade prepare

This command performs the initial preparation steps for the overcloud upgrade, which includes replacing the current overcloud plan on the undercloud with the new OpenStack Platform 16.1 overcloud plan and your updated environment files. This command functions similar to the openstack overcloud deploy command and uses many of the same options.

15.2. openstack overcloud upgrade run

This command performs the upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 16.1 overcloud plan and runs the fast forward tasks on the entire overcloud. This includes running the upgrade process through each OpenStack Platform version from 13 to 16.1.

In addition to the standard upgrade process, this command can perform a Leapp upgrade of the operating system on overcloud nodes. Run these tasks using the --tags option.

Upgrade task tags for Leapp

system_upgrade
Task that combines tasks from system_upgrade_prepare, system_upgrade_run, and system_upgrade_reboot.
system_upgrade_prepare
Tasks to prepare for the operating system upgrade with Leapp.
system_upgrade_run
Tasks to run Leapp and upgrade the operating system.
system_upgrade_reboot
Tasks to reboot a system and complete the operating system upgrade.

Upgrade task tags for workload migration

nova_hybrid_state
Task that sets up temporary OpenStack Platform 16.1 containers on Compute nodes to facilitate workload migration during the upgrade.

15.3. openstack overcloud external-upgrade run

This command performs upgrade tasks outside the standard upgrade process. Director creates a set of Ansible playbooks based on the new OpenStack Platform 16.1 overcloud plan and you run specific tasks using the --tags option.

External task tags for container management

container_image_prepare
Tasks for pulling container images to the undercloud registry and preparing the images for the overcloud to use.

External task tags for Ceph Storage upgrades

ceph
Tasks to install Red Hat Ceph Storage using ceph-ansible playbooks.
ceph_systemd
Tasks to convert Red Hat Ceph Storage systemd unit files to use podman management.

External task tags for database transfer

system_upgrade_cleanup
Tasks to clean storage directories related to system_upgrade_transfer_data tasks.
system_upgrade_stop_services
Tasks to shut down all services.
system_upgrade_transfer_data
Tasks to shut down all services and perform a database transfer to the bootstrap node.

15.4. openstack overcloud upgrade converge

This command performs the final step in the overcloud upgrade. This final step synchronizes the overcloud heat stack with the OpenStack Platform 16.1 overcloud plan and your updated environment files. This process ensures that the resulting overcloud matches the configuration of a new OpenStack Platform 16.1 overcloud. This command is similar to the openstack overcloud deploy command and uses many of the same options.

15.5. Overcloud node upgrade workflow

When you perform an upgrade on each overcloud node, you must consider the following aspects to determine the correct commands to run at the relevant stage in the upgrade:

Controller Services

  • Does the node contain Pacemaker services? You must first upgrade the bootstrap node, which includes a database transfer and launching temporary containers to facilitate migration during the transition from Red Hat OpenStack 13 to 16.1. After upgrading the bootstrap node, upgrade each additional node with pacemaker services and ensure the each node joins the new Pacemaker cluster started with the bootstrap node. The process for upgrading split-service Controller nodes without Pacamaker does not require these additional steps.

Compute Services

  • Is the node a Compute node? If the node does contain Compute services, you must migrate virtual machines from the node to ensure maximum availability. A Compute node in this situation includes any node designed to host virtual machines. This definition includes the follow Compute nodes types:

    • Regular Compute nodes
    • Compute nodes with Hyper-Converged Infrastructure (HCI)
    • Compute nodes with Network Function Virtualization technologies such as Data Plane Development Kit (DPDK) or Single Root Input/Output Virtualization (SR-IOV)
    • Real Time Compute nodes

Ceph Storage Services

  • Does the node contain any Ceph Storage services? You must convert the systemd unit files for any containerized Ceph Storage services on the node to use podman instead of docker. This applies to the following node types:

    • Ceph Storage OSD nodes
    • Controller nodes with Ceph MON services
    • Split-Controller Ceph MON nodes
    • Compute nodes with Hyper-Converged Infrastructure (HCI)

Workflow

Use the following workflow diagram to identify the correct update path for specific nodes:

Chapter 16. Upgrading a standard overcloud

This scenario contains an example upgrade process for a standard overcloud environment, which includes the following node types:

  • Three Controller nodes
  • Three Ceph Storage nodes
  • Multiple Compute nodes

16.1. Running the overcloud upgrade preparation

The upgrade requires running openstack overcloud upgrade prepare command, which performs the following tasks:

  • Updates the overcloud plan to OpenStack Platform 16.1
  • Prepares the nodes for the upgrade
Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade preparation command:

    $ openstack overcloud upgrade prepare \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  3. Wait until the upgrade preparation completes.
  4. Download the container images:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

16.2. Upgrading Controller nodes

Upgrade all the Controller nodes to OpenStack Platform 16.1. This process involves upgrading each Controller node starting with the bootstrap Controller node.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Identify the bootstrap Controller node by running the following command on a Controller node:

    $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"
  3. Upgrade the bootstrap Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

    3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

    4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

    5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

  4. Upgrade the next Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

  5. Upgrade the final Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

16.3. Upgrading the operating system for Ceph Storage nodes

Upgrade the operating system for each Ceph Storage nodes.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Log in to a node with Ceph MON services:
  3. Disable OSD exclusion and rebalancing:

    $ sudo docker ps | grep ceph-mon
    $ sudo docker exec -it CONTAINER ceph osd set noout
    $ sudo docker exec -it CONTAINER ceph osd set norebalance

    Replace CONTAINER with the name of the container running Ceph MON.

  4. Log out of the node with Ceph MON services and return to the undercloud.
  5. Select a Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  6. Select the next Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  7. Select the final Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.
  9. Enable OSD exclusion and rebalancing:

    $ sudo podman ps | grep ceph-mon
    $ sudo podman exec -it CONTAINER ceph osd unset noout
    $ sudo podman exec -it CONTAINER ceph osd unset norebalance

    Replace CONTAINER with the name of the container running Ceph MON.

  10. Log out of the node with Ceph MON services and return to the undercloud.

16.4. Upgrading Compute nodes

Upgrade all the Compute nodes to OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"
  3. Run the upgrade command with the system_upgrade tag:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0 --playbook upgrade_steps_playbook.yaml

    This command performs the following actions:

    • Performs a Leapp upgrade of the operating system.
    • Performs a reboot as a part of the Leapp upgrade.
  4. Run the upgrade command with no tags:

    $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

    This command performs the Red Hat OpenStack Platform upgrade.

  5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2 --playbook upgrade_steps_playbook.yaml

    Then perform the standard OpenStack service upgrade:

    $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

16.5. Synchronizing the overcloud stack

The upgrade requires an update the overcloud stack to ensure that the stack resource structure and parameters align with a fresh deployment of OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

    • ceph3_namespace
    • ceph3_tag
    • ceph3_image
    • name_prefix_stein
    • name_suffix_stein
    • namespace_stein
    • tag_stein
  3. Run the upgrade finalization command:

    $ openstack overcloud upgrade converge \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  4. Wait until the stack synchronization completes.
Important

You do not need the upgrades-environment.yaml file for any further deployment operations.

16.6. Upgrading to Ceph Storage 4

Upgrade the Ceph Storage nodes from version 3 to version 4.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the Ceph Storage external upgrade process with the ceph tag:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph
  3. Wait until the Ceph Storage upgrade completes.

Chapter 17. Upgrading a split Controller overcloud

This scenario contains an example upgrade process for an overcloud with Controller node services split on to multiple nodes. This includes the following node types:

  • Multiple split high availability services using Pacemaker
  • Multiple split Controller services
  • Three Ceph MON nodes
  • Three Ceph Storage nodes
  • Multiple Compute nodes

17.1. Running the overcloud upgrade preparation

The upgrade requires running openstack overcloud upgrade prepare command, which performs the following tasks:

  • Updates the overcloud plan to OpenStack Platform 16.1
  • Prepares the nodes for the upgrade
Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade preparation command:

    $ openstack overcloud upgrade prepare \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  3. Wait until the upgrade preparation completes.
  4. Download the container images:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

17.2. Upgrading Pacemaker-based nodes

Upgrade all nodes that host Pacemaker services to OpenStack Platform 16.1. The following roles include Pacemaker-based services:

  • Controller
  • Database (MySQL, Galera)
  • Messaging (RabbitMQ)
  • Load Balancing (HAProxy)
  • Any other role that contains the following services:

    • OS::TripleO::Services::Pacemaker
    • OS::TripleO::Services::PacemakerRemote

This process involves upgrading each node starting with the bootstrap node.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Identify the bootstrap node by running the following command on a Controller-based node:

    $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"
  3. Upgrade the bootstrap node:

    1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

    4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

    5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

  4. Upgrade each Pacemaker-based node:

    1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-database-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag on the next node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-database-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-database-0

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include any previously upgraded node in the --limit option.

  5. Repeat the upgrade process on each Pacemaker-based node until you have upgraded all Pacemaker-based node.

17.3. Upgrading non-Pacemaker Controller nodes

Upgrade all nodes without Pacemaker-based services to OpenStack Platform 16.1. These nodes usually contain a specific OpenStack service. Examples of roles without Pacemaker-based services include the following:

  • Networker
  • Ironic Conductor
  • Object Storage
  • Any custom roles with services split or scaled from standard Controller nodes

Do not include the following nodes in this grouping:

  • Any Compute nodes
  • Any Ceph Storage nodes

This process involves upgrading each node.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command with the system_upgrade tag:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-networker-0 --playbook upgrade_steps_playbook.yaml

    This command performs the following actions:

    • Performs a Leapp upgrade of the operating system.
    • Performs a reboot as a part of the Leapp upgrade.
  3. Run the upgrade command with no tags:

    $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-networker-0

    This command performs the Red Hat OpenStack Platform upgrade.

  4. Repeat the upgrade process on each node until you have upgraded all Controller-based node.

17.4. Upgrading the operating system for Ceph MON nodes

Upgrade the operating system for each Ceph MON node. It is recommended to upgrade each Ceph MON node individually to maintain a quorum among the nodes.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Select a Ceph MON node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-0

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  3. Select the next Ceph MON node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-1

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  4. Select the final Ceph MON node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-2

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

17.5. Upgrading the operating system for Ceph Storage nodes

Upgrade the operating system for each Ceph Storage nodes.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Log in to a node with Ceph MON services:
  3. Disable OSD exclusion and rebalancing:

    $ sudo docker ps | grep ceph-mon
    $ sudo docker exec -it CONTAINER ceph osd set noout
    $ sudo docker exec -it CONTAINER ceph osd set norebalance

    Replace CONTAINER with the name of the container running Ceph MON.

  4. Log out of the node with Ceph MON services and return to the undercloud.
  5. Select a Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  6. Select the next Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  7. Select the final Ceph Storage node and upgrade the operating system:

    1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

  8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.
  9. Enable OSD exclusion and rebalancing:

    $ sudo podman ps | grep ceph-mon
    $ sudo podman exec -it CONTAINER ceph osd unset noout
    $ sudo podman exec -it CONTAINER ceph osd unset norebalance

    Replace CONTAINER with the name of the container running Ceph MON.

  10. Log out of the node with Ceph MON services and return to the undercloud.

17.6. Upgrading Compute nodes

Upgrade all the Compute nodes to OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"
  3. Run the upgrade command with the system_upgrade tag:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0 --playbook upgrade_steps_playbook.yaml

    This command performs the following actions:

    • Performs a Leapp upgrade of the operating system.
    • Performs a reboot as a part of the Leapp upgrade.
  4. Run the upgrade command with no tags:

    $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

    This command performs the Red Hat OpenStack Platform upgrade.

  5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2 --playbook upgrade_steps_playbook.yaml

    Then perform the standard OpenStack service upgrade:

    $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

17.7. Synchronizing the overcloud stack

The upgrade requires an update the overcloud stack to ensure that the stack resource structure and parameters align with a fresh deployment of OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

    • ceph3_namespace
    • ceph3_tag
    • ceph3_image
    • name_prefix_stein
    • name_suffix_stein
    • namespace_stein
    • tag_stein
  3. Run the upgrade finalization command:

    $ openstack overcloud upgrade converge \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  4. Wait until the stack synchronization completes.
Important

You do not need the upgrades-environment.yaml file for any further deployment operations.

17.8. Upgrading to Ceph Storage 4

Upgrade the Ceph Storage nodes from version 3 to version 4.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the Ceph Storage external upgrade process with the ceph tag:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph
  3. Wait until the Ceph Storage upgrade completes.

Chapter 18. Upgrading an overcloud with hyper-converged infrastructure

This scenario contains an example upgrade process for an overcloud with hyper-converged infrastructure (HCI), which includes the following node types:

  • Three Controller nodes
  • Multiple HCI Compute nodes, which contain combined Compute and Ceph OSD services

18.1. Running the overcloud upgrade preparation

The upgrade requires running openstack overcloud upgrade prepare command, which performs the following tasks:

  • Updates the overcloud plan to OpenStack Platform 16.1
  • Prepares the nodes for the upgrade
Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade preparation command:

    $ openstack overcloud upgrade prepare \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  3. Wait until the upgrade preparation completes.
  4. Download the container images:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

18.2. Upgrading Controller nodes

Upgrade all the Controller nodes to OpenStack Platform 16.1. This process involves upgrading each Controller node starting with the bootstrap Controller node.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Identify the bootstrap Controller node by running the following command on a Controller node:

    $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"
  3. Upgrade the bootstrap Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

    3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

    4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

    5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

  4. Upgrade the next Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

  5. Upgrade the final Controller node:

    1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

    2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.
    3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

18.3. Upgrading Compute nodes with hyper-converged infrastructure (HCI)

Upgrade HCI Compute nodes to OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"
  3. Log in to a node with Ceph MON services:
  4. Disable OSD exclusion and rebalancing:

    $ sudo docker ps | grep ceph-mon
    $ sudo docker exec -it CONTAINER ceph osd set noout
    $ sudo docker exec -it CONTAINER ceph osd set norebalance

    Replace CONTAINER with the name of the container that runs Ceph MON.

  5. Log out of the node with Ceph MON services and return to the undercloud.
  6. Run the external upgrade command with the ceph_systemd tag:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

    This command performs the following functions:

    • Changes the systemd units that control the Ceph Storage containers to use Podman management.
    • Limits actions to the selected Ceph Storage node using the ceph_ansible_limit variable.

    This step is a preliminary measure to prepare the Ceph Storage services for the leapp upgrade.

  7. Run the upgrade command with the system_upgrade tag:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0 --playbook upgrade_steps_playbook.yaml

    This command performs the following actions:

    • Performs a Leapp upgrade of the operating system.
    • Performs a reboot as a part of the Leapp upgrade.
  8. Run the upgrade command with no tags:

    $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-computehci-0

    This command performs the Red Hat OpenStack Platform upgrade.

  9. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First run the external upgrade command with the ceph_systemd tag:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

    Then perform the system_upgrade task:

    $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2 --playbook upgrade_steps_playbook.yaml

    Then perform the standard OpenStack service upgrade:

    $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-computehci-1,overcloud-computehci-2
  10. After you have upgraded all HCI nodes, log in to a node with Ceph MON services.
  11. Enable OSD exclusion and rebalancing:

    $ sudo podman ps | grep ceph-mon
    $ sudo podman exec -it CONTAINER ceph osd unset noout
    $ sudo podman exec -it CONTAINER ceph osd unset norebalance

    Replace CONTAINER with the name of the container that runs Ceph MON.

  12. Log out of the node with Ceph MON services and return to the undercloud.

18.4. Synchronizing the overcloud stack

The upgrade requires an update the overcloud stack to ensure that the stack resource structure and parameters align with a fresh deployment of OpenStack Platform 16.1.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

    • ceph3_namespace
    • ceph3_tag
    • ceph3_image
    • name_prefix_stein
    • name_suffix_stein
    • namespace_stein
    • tag_stein
  3. Run the upgrade finalization command:

    $ openstack overcloud upgrade converge \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        -e /home/stack/templates/rhsm.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
        -e /home/stack/templates/workarounds.yaml \
        …​

    Include the following options relevant to your environment:

    • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
    • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
    • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
    • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
    • The temporary workarounds file (workarounds.yaml).
    • Any custom configuration environment files (-e) relevant to your deployment.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • If applicable, your composable network (network_data) file using --networks-file.
    • If you use a custom stack name, pass the name with the --stack option.
  4. Wait until the stack synchronization completes.
Important

You do not need the upgrades-environment.yaml file for any further deployment operations.

18.5. Upgrading to Ceph Storage 4

Upgrade the Ceph Storage nodes from version 3 to version 4.

Note

If you are not using the default stack name (overcloud), set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the Ceph Storage external upgrade process with the ceph tag:

    $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph
  3. Wait until the Ceph Storage upgrade completes.

Part IV. Concluding the overcloud upgrade

Chapter 19. Performing post-upgrade actions

After you have completed the overcloud upgrade, you must perform some post-upgrade configuration to ensure that your environment is fully supported and ready for future operations.

19.1. Validating the post-upgrade functionality

Run the post-upgrade validation group to check the post-upgrade functionality.

Procedure

  1. Source the stackrc file.

    $ source ~/stackrc
  2. Run the openstack tripleo validator run command with the --group post-upgrade option:

    $ openstack tripleo validator run --group post-upgrade
  3. Review the results of the validation report.
Important

A FAILED validation does not prevent you from deploying or running Red Hat OpenStack Platform. However, a FAILED validation can indicate a potential issue with a production environment.

19.2. Upgrading the overcloud images

You must replace your current overcloud images with new versions. The new images ensure that the director can introspect and provision your nodes using the latest version of OpenStack Platform software.

Prerequisites

  • You have upgraded the undercloud to the latest version.

Procedure

  1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

    $ rm -rf ~/images/*
  2. Extract the archives:

    $ cd ~/images
    $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done
    $ cd ~
  3. Import the latest images into the director:

    $ openstack overcloud image upload --update-existing --image-path /home/stack/images/
  4. Configure your nodes to use the new images:

    $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)
Important

When you deploy overcloud nodes, ensure that the overcloud image version corresponds to the respective heat template version. For example, use the OpenStack Platform 16.1 images only with the OpenStack Platform 16.1 heat templates.

Important

The new overcloud-full image replaces the old overcloud-full image. If you made changes to the old image, you must repeat the changes in the new image, especially if you want to deploy new nodes in the future.

19.3. Migrating to Open Virtual Network (OVN)

Open Virtual Network (OVN) is an Open vSwitch-based software-defined networking (SDN) solution for supplying network services to instances. OVN provides platform-agnostic support for the full OpenStack Networking API. You can use OVN to connect groups of guest instances programatically into private L2 and L3 networks. OVN uses a standard approach to virtual networking that can extend to other Red Hat platforms and solutions.

After you upgrade to Red Hat Enterprise Linux 16.1, migrate the OpenStack Networking (neutron) service on your overcloud from the ML2/OVS driver to the ML2/OVN driver. For more information on migrating to ML2/OVN, see "Migrating from ML2/OVS to ML2/OVN".

Part V. Troubleshooting

Chapter 20. Troubleshooting upgrade issues

If you experience any issues with during the upgrade process, refer to the advice in this section.

20.1. Correcting environment files

If you have made a mistake with any parameters in any custom environment files, you can correct the environment file and run the openstack overcloud upgrade prepare command at any time during the upgrade. This command uploads a new version of your overcloud plan to director, which will generate a new set of config-download playbooks.

This example contains a repository name mistake in the upgrades-environment.yaml file:

parameter_defaults:
  UpgradeLeappEnabled: true
  UpgradeLeappCommandOptions: "--enablerepo rhel-7-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
  CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms

This mistake causes an issue during the Leapp upgrade for the Controller node. To rectify this issue, correct the mistake and run the openstack overcloud upgrade prepare command.

Procedure

  1. Correct the mistake in the file:

    parameter_defaults:
      UpgradeLeappEnabled: true
      UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
      CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
  2. Run the upgrade preparation command with the corrected file:

    $ openstack overcloud upgrade prepare \
        --stack STACK NAME \
        --templates \
        -e ENVIRONMENT FILE
        …​
        -e /home/stack/templates/upgrades-environment.yaml \
        …​

    Wait until the overcloud stack update completes.

  3. Continue with the upgrade operation step that failed.