Procedure

1. Log in to the undercloud as the stack user.

2. Create an Ansible playbook named playbook-nics.yaml and copy the following content into the playbook:

   ---
   - name: Rename eth devices
     hosts: all
     become: yes
     vars:
       prefix: "em"
     tasks:
       - name: Update udev rules
         lineinfile:
           line: "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"{{ ansible_facts[item]['perm_macaddress'] | default(ansible_facts[item]['macaddress']) }}\", NAME=\"{{ prefix }}{{ item|replace('eth','') }}\""
           path: /etc/udev/rules.d/70-rhosp-persistent-net.rules
           create: True
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")
       - name: Copy network-script files using the new prefix
         copy:
           remote_src: True
           src: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
           dest: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")
       - name: Edit NAME in new network-script files
         lineinfile:
           regexp: "^NAME=.*"
           line: "NAME={{ prefix }}{{ item|replace('eth','') }}"
           path: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")
       - name: Edit DEVICE in new network-script files
         lineinfile:
           regexp: "^DEVICE=.*"
           line: "DEVICE={{ prefix }}{{ item|replace('eth','') }}"
           path: /etc/sysconfig/network-scripts/ifcfg-{{ prefix }}{{ item|replace('eth','') }}
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")
       - name: Backup old eth network-script files
         copy:
           remote_src: True
           src: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
           dest: /etc/sysconfig/network-scripts/ifcfg-{{ item }}.bak
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")
       - name: Backup old eth network-script files
         file:
           path: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
           state: absent
         with_items: "{{ ansible_interfaces }}"
         when: item.startswith("eth")

   Note

   You will use this playbook to rename the overcloud NICs at a later stage in the upgrade process.

3. Run the playbook-nics.yaml playbook on the undercloud:

   $ ansible-playbook -c local -i localhost, playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -c local -i localhost, -e prefix="mynic" ~/playbook-nics.yaml

4. Reboot the undercloud node using the standard reboot procedures. For more information, see "Rebooting nodes".

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

   $ sudo grep PermitRootLogin /etc/ssh/sshd_config

3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

   PermitRootLogin no

4. Save the file.

Procedure

1. Log in to the undercloud as the stack user.

2. Disable the main OpenStack services on the undercloud:

   $ sudo systemctl stop openstack-* httpd haproxy mariadb rabbitmq* docker xinetd

3. Remove the main OpenStack services from the undercloud, except OpenvSwitch and certain Python 2 packages that are required for the upgrade:

   $ sudo yum -y remove *el7ost* galera* haproxy* \
       httpd mysql* pacemaker* xinetd python-jsonpointer \
       qemu-kvm-common-rhev qemu-img-rhev rabbit* \
       redis* \
       -- \
       -*openvswitch* -python-docker -python-PyMySQL \
       -python-pysocks -python2-asn1crypto -python2-babel \
       -python2-cffi -python2-cryptography -python2-dateutil \
       -python2-idna -python2-ipaddress -python2-jinja2 \
       -python2-jsonpatch -python2-markupsafe -python2-pyOpenSSL \
       -python2-requests -python2-six -python2-urllib3

4. Remove the content from the /etc/httpd and /var/lib/docker directories:

   $ sudo rm -rf /etc/httpd /var/lib/docker

Procedure

1. Log in to the undercloud as the stack user.

2. Install the Leapp utility:

   $ sudo yum install leapp

3. Download the additional required data files (RPM package changes and RPM repository mapping) attached to the Knowledge Base article "Data required by the Leapp utility for an in-place upgrade from RHEL 7 to RHEL 8" and place these files in the /etc/leapp/files/ directory.

4. Update your Red Hat subscription:

   • If your undercloud uses the Red Hat Customer Portal for registration, refresh your current subscription to obtain access to the Red Hat Enterprise Linux 8.2 content:

     $ sudo subscription-manager refresh

   • If your undercloud uses a Red Hat Satellite server for registration, re-register the undercloud to a content view associated with your Red Hat OpenStack Platform 16.1 activation key.

     $ subscription-manager register --force --org ORG --activationkey ACTIVATION_KEY --release 8.2

     Note

     The content view that you create for Red Hat OpenStack Platform 16.1 should contain content for Red Hat Enterprise Linux 8.2.

5. Red Hat OpenStack Platform 16.1 uses a newer version of Open vSwitch`. Substitute the Open vSwitch version through the to_remove and to_install transaction files:

   $ echo 'openvswitch2.11' | sudo tee -a /etc/leapp/transaction/to_remove
   $ echo 'openvswitch2.13' | sudo tee -a /etc/leapp/transaction/to_install

6. Start the Leapp upgrade process:

   $ sudo leapp upgrade --debug --enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms

   Use the --enablerepo option to set the repositories that you want to enable during the Leapp upgrade process. You must include these repositories to facilitate the Red Hat OpenStack Platform 16.1 transition, especially with the newer version of Open vSwitch.

7. Wait for the leapp upgrade command to successfully complete.

8. Create an empty .autorelabel file in your root directory:

   $ sudo touch /.autorelabel

   After a reboot, SELinux detects this file and automatically relabels the file system.

9. Reboot the undercloud:

   $ sudo reboot

Procedure

1. Remove the unnecessary packages

   $ sudo dnf -y remove python2*

Procedure

1. Log in to the undercloud as the stack user.

2. Lock the undercloud to a specific verison with the subscription-manager release command:

   $ sudo subscription-manager release --set=8.2

Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

   These repositories contain packages that the director installation requires.

3. Synchronize the operating system to ensure that your system packages match the operating system version:

   [stack@director ~]$ sudo dnf distro-sync -y
   [stack@director ~]$ sudo reboot

Procedure

1. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. With this option, director pulls the necessary images from the Red Hat Container Catalog and pushes the images to the registry on the undercloud. Director uses the undercloud registry as the container image source. To pull container images directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file specifies an environment file that includes include the parameters for preparing your container images. In this example, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Modify the containers-prepare-parameter.yaml to suit your requirements.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the containers-prepare-parameter.yaml file.

3. Add the following parameters to the set in the ContainerImagePrepare parameter:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ...
         name_prefix_stein: openstack-
         name_suffix_stein: ''
         namespace_stein: registry.redhat.io/rhosp15-rhel8
         tag_stein: 15.0
         ceph3_namespace: registry.redhat.io/rhceph
         ceph3_tag: latest
         ceph3_image: rhceph-3-rhel7
         ...

   • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
   • The ceph3_* parameters define the current Red Hat Ceph Storage container images that the overcloud uses. The overcloud requires these containers until the transition from Red Hat Ceph Storage 3 to 4.
   • If you use Red Hat Satellite for container image storage, set the namespaces to the image locations on the Red Hat Satellite server.

4. Change the neutron_driver parameter to openvswitch:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ...
         neutron_driver: openvswitch
         ...

   The upgrade retains Open vSwitch compatibility throughout the process. After you complete the upgrade to Red Hat OpenStack Platform 16.1, you migrate the overcloud from Open vSwitch to Open Virtual Network (OVN).

5. Save the containers-prepare-parameter.yaml file.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the undercloud.conf file.

3. Add the following parameter to the DEFAULT section in the file:

   container_images_file = /home/stack/containers-prepare-parameter.yaml

   This parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.

4. Check the local_interface parameter if you have migrated to a predictable NIC naming convention.
5. Check all other parameters in the file for any changes.
6. Save the file.

Procedure

1. Run the following command to upgrade the director on the undercloud:

   $ openstack undercloud upgrade

   This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

   Note

   The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the -y option:

   $ openstack undercloud upgrade -y

2. The script also starts all OpenStack Platform service containers on the undercloud automatically. You manage each service through a systemd resource. Check the systemd resources:

   $ sudo systemctl list-units "tripleo_*"

   Each systemd service controls a container. Check the enabled containers using the following command:

   $ sudo podman ps

3. The script adds the stack user to the docker group to ensure that the stack user has access to container management commands. Refresh the stack user permissions with the following command:

   $ exec su -l stack

   The command prompts you to log in again. Enter the stack user password.

4. To initialize the stack user to use the command line tools, run the following command:

   $ source ~/stackrc

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

   (undercloud) $

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group pre-upgrade option:

   $ openstack tripleo validator run --group pre-upgrade

3. Review the results of the validation report.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Create a static inventory file of all nodes:

   $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml

4. To execute Ansible playbooks on your environment, run the ansible-playbook command and include the full path of the dynamic inventory tool using the -i option. For example:

   (undercloud) $ ansible-playbook -i ~/inventory.yaml PLAYBOOK

Procedure

1. Log in to the undercloud as the stack user.

2. Create an environment file called upgrades-environment.yaml in your templates directory:

   $ touch templates/upgrades-environment.yaml

3. Edit the file and add the following mandatory content:

   parameter_defaults:
     UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms

   • UpgradeLeappCommandOptions sets additional options to pass to the Leapp tool. In this context, set this parameter to enable Red Hat OpenStack Platform repositories to help with the Leapp transition.
   • CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.
4. Save the upgrades-environment.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-leapp-data.yaml and paste the following content in the file:

   ---
   - name: Copy Leapp data files to overcloud
     hosts: overcloud
     become: yes
     tasks:
       - name: Create the Leapp files directory
         file:
           path: /etc/leapp/files
           state: directory
       - name: Copy pes-events.json
         copy:
           src: /etc/leapp/files/pes-events.json
           dest: /etc/leapp/files/pes-events.json
       - name: Copy
         copy:
           src: /etc/leapp/files/repomap.csv
           dest: /etc/leapp/files/repomap.csv

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-leapp-data.yaml

Procedure

1. Log in to the undercloud as the stack user.

2. Run the playbook-nics.yaml playbook on all overcloud nodes:

   $ ansible-playbook -i ~/inventory.yaml playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -i ~/inventory.yaml -e prefix="mynic" playbook-nics.yaml

3. Reboot overcloud nodes using the standard reboot procedures. For more information, see "Rebooting nodes".

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-ssh.yaml and paste the following content in the file:

   ---
   - name: Configure SSH PermitRootLogin parameter
     hosts: overcloud
     become: yes
     tasks:
       - name: Set the PermitRootLogin parameter to no
         lineinfile:
           path: /etc/ssh/sshd_config
           state: present
           line: "PermitRootLogin no"

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-ssh.yaml

Procedure

1. Log in to the undercloud as the stack user.

2. Obtain the control plane host name on the undercloud:

   $ sudo hiera container_image_prepare_node_names
   ["undercloud.ctlplane.localdomain"]

3. Edit the containers-prepare-parameter.yaml file and add the DockerInsecureRegistryAddress parameter with a YAML list that contains the undercloud control plane host name:

   parameter_defaults:
     DockerInsecureRegistryAddress:
     - undercloud.ctlplane.localdomain:8787
     ...

   You must also append the port number of the overcloud registry to the host name value. The port number is 8787.

Procedure

1. Search for custom environment files that use composable services. You usually store custom environment files in the /home/stack/templates directory:

   $ cd ~/templates/
   $ grep "OS::TripleO::Services" *

   In this scenario, one of the files shows an outdated mapping:

     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mgr.yaml

2. Identify the new ceph-mgr.yaml location in /usr/share/openstack-tripleo-heat-templates/. This file is now located in the `deployment/ceph-ansible' directory:

   $ find /usr/share/openstack-tripleo-heat-templates/ -name ceph-mgr.yaml
   /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

3. Edit the service in the custom environment file:

   resource_registry:
     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

   Save the file.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file (templates/rhsm.yml) to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-8-for-x86_64-baseos-eus-rpms
         - rhel-8-for-x86_64-appstream-eus-rpms
         - rhel-8-for-x86_64-highavailability-eus-rpms
         - ansible-2.9-for-rhel-8-x86_64-rpms
         - advanced-virt-for-rhel-8-x86_64-rpms
         - openstack-16.1-for-rhel-8-x86_64-rpms
         - rhceph-4-osd-for-rhel-8-x86_64-rpms
         - rhceph-4-mon-for-rhel-8-x86_64-rpms
         - rhceph-4-tools-for-rhel-8-x86_64-rpms
         - fast-datapath-for-rhel-8-x86_64-rpms
       rhsm_username: "myusername"
       rhsm_password: "p@55w0rd!"
       rhsm_org_id: "1234567"
       rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
       rhsm_method: "portal"
       rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

3. Save the environment file.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file (templates/rhsm.yml) to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_activation_key: "myactivationkey"
       rhsm_method: "satellite"
       rhsm_org_id: "ACME"
       rhsm_server_hostname: satellite.example.com"
       rhsm_baseurl: "https://satellite.example.com/pulp/repos"
       rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

3. Save the environment file.

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-satellite.yaml and paste the following content in the file:

   - name: Pre-install leapp
     hosts: overcloud
     become: yes
     tasks:
       - name: Pre-install leapp
         yum:
           name:
             - leapp
             - leapp-repository
           state: installed
       - name: Remove katello-host-tools-fact-plugin
         yum:
           name:
             - katello-host-tools-fact-plugin
           state: removed
       - name: Register system
         import_role:
           name: redhat-subscription
         vars:
           rhsm_activation_key: "osp16-key"
           rhsm_method: "satellite"
           rhsm_org_id: "ACME"
           rhsm_server_hostname: "satellite.example.com"
           rhsm_baseurl: "https://satellite.example.com/pulp/repos"
           rhsm_release: "8.2"
           rhsm_force_register: True

   Modify the rhsm_* variables to suit your Satellite server.

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-satellite.yaml

Procedure

1. Run the following script on the undercloud:

   STACK_NAME="overcloud"
   ROLES_DATA="/usr/share/openstack-tripleo-heat-templates/roles_data.yaml"
   NIC_CONFIG_LINES=$(openstack stack environment show $STACK_NAME | grep "::Net::SoftwareConfig" | sed -E 's/ *OS::TripleO::// ; s/::Net::SoftwareConfig:// ; s/ http.*user-files/ /')
   echo "$NIC_CONFIG_LINES"
   echo "$NIC_CONFIG_LINES" | while read LINE; do
       ROLE=$(echo "$LINE" | awk '{print $1;}')
       NIC_CONFIG=$(echo "$LINE" | awk '{print $2;}')
   
       python3 /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
               --tht-dir /usr/share/openstack-tripleo-heat-templates \
               --roles-data $ROLES_DATA \
               --role-name "$ROLE" \
               --discard-comments yes \
               --template "$NIC_CONFIG"
   done

   • If you use a custom overcloud name, Set the STACK_NAME variable to the name of your overcloud. The default name for an overcloud stack is overcloud.
   • If you use a custom roles_data file, set the ROLES_DATA variable to the location of the custom file. If you use the default roles_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
   • Run /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py -h to see a list of options to add to the script.

Procedure

1. Log in to the undercloud as the stack user.

2. Edit the workarounds.yaml file and add the UpgradeLevelNovaCompute parameter to the parameter_defaults section:

   parameter_defaults:
     UpgradeLevelNovaCompute: ''

3. Save the workarounds.yaml file.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on a Controller node:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

3. Upgrade the bootstrap Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

4. Upgrade the next Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Log in to a node with Ceph MON services:

3. Disable OSD exclusion and rebalancing:

   $ sudo docker ps | grep ceph-mon
   $ sudo docker exec -it CONTAINER ceph osd set noout
   $ sudo docker exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

4. Log out of the node with Ceph MON services and return to the undercloud.

5. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

6. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

7. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.

9. Enable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd unset noout
   $ sudo podman exec -it CONTAINER ceph osd unset norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

10. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0 --playbook upgrade_steps_playbook.yaml

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2 --playbook upgrade_steps_playbook.yaml

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph

3. Wait until the Ceph Storage upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap node by running the following command on a Controller-based node:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

3. Upgrade the bootstrap node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

4. Upgrade each Pacemaker-based node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-database-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-database-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-database-0

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include any previously upgraded node in the --limit option.

5. Repeat the upgrade process on each Pacemaker-based node until you have upgraded all Pacemaker-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-networker-0 --playbook upgrade_steps_playbook.yaml

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

3. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-networker-0

   This command performs the Red Hat OpenStack Platform upgrade.

4. Repeat the upgrade process on each node until you have upgraded all Controller-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Select a Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-0

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-1

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-2

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Log in to a node with Ceph MON services:

3. Disable OSD exclusion and rebalancing:

   $ sudo docker ps | grep ceph-mon
   $ sudo docker exec -it CONTAINER ceph osd set noout
   $ sudo docker exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

4. Log out of the node with Ceph MON services and return to the undercloud.

5. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

6. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

7. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.

9. Enable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd unset noout
   $ sudo podman exec -it CONTAINER ceph osd unset norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

10. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0 --playbook upgrade_steps_playbook.yaml

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2 --playbook upgrade_steps_playbook.yaml

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph

3. Wait until the Ceph Storage upgrade completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on a Controller node:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

3. Upgrade the bootstrap Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

4. Upgrade the next Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. If your overcloud includes a director-deployed Ceph Storage cluster, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2 --playbook upgrade_steps_playbook.yaml

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"
3. Log in to a node with Ceph MON services:

4. Disable OSD exclusion and rebalancing:

   $ sudo docker ps | grep ceph-mon
   $ sudo docker exec -it CONTAINER ceph osd set noout
   $ sudo docker exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container that runs Ceph MON.

5. Log out of the node with Ceph MON services and return to the undercloud.

6. Run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

   This command performs the following functions:

   • Changes the systemd units that control the Ceph Storage containers to use Podman management.
   • Limits actions to the selected Ceph Storage node using the ceph_ansible_limit variable.

   This step is a preliminary measure to prepare the Ceph Storage services for the leapp upgrade.

7. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0 --playbook upgrade_steps_playbook.yaml

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

8. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-computehci-0

   This command performs the Red Hat OpenStack Platform upgrade.

9. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

   Then perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2 --playbook upgrade_steps_playbook.yaml

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-computehci-1,overcloud-computehci-2

10. After you have upgraded all HCI nodes, log in to a node with Ceph MON services.

11. Enable OSD exclusion and rebalancing:

    $ sudo podman ps | grep ceph-mon
    $ sudo podman exec -it CONTAINER ceph osd unset noout
    $ sudo podman exec -it CONTAINER ceph osd unset norebalance

    Replace CONTAINER with the name of the container that runs Ceph MON.

12. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       -e /home/stack/templates/workarounds.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • The temporary workarounds file (workarounds.yaml).
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph

3. Wait until the Ceph Storage upgrade completes.

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group post-upgrade option:

   $ openstack tripleo validator run --group post-upgrade

3. Review the results of the validation report.

Procedure

1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

2. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done
   $ cd ~

3. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

4. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

Procedure

1. Correct the mistake in the file:

   parameter_defaults:
     UpgradeLeappEnabled: true
     UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Run the upgrade preparation command with the corrected file:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       …​

   Wait until the overcloud stack update completes.

3. Continue with the upgrade operation step that failed.