Procedure

1. Log in to the undercloud as the stack user.

2. Create an Ansible playbook named playbook-nics.yaml and copy the following content into the playbook:

   ---
   - name: Rename eth devices
     hosts: all
     become: yes
     vars:
       prefix: "em"
     tasks:
       - set_fact:
           eth_interfaces: "{{ ansible_interfaces | select('match','eth.*') | list }}"
       - debug:
           msg: "{{ eth_interfaces }}"
       - name: Update udev rules
         lineinfile:
           line: "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"{{ ansible_facts[item]['perm_macaddress'] | default(ansible_facts[item]['macaddress']) }}\", NAME=\"{{ item|replace('eth',prefix) }}\""
           path: /etc/udev/rules.d/70-rhosp-persistent-net.rules
           create: True
         with_items: "{{ eth_interfaces }}"
       - name: Rename eth files
         block:
           - name: Check that eth files exists
             stat:
               path: /etc/sysconfig/network-scripts/ifcfg-{{ item }}
             register: nic_result
             with_items: "{{ eth_interfaces }}"
           - name: Copy nic files using the new prefix
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Edit NAME in new network-script files
             lineinfile:
               regexp: "^NAME=.*"
               line: "NAME={{ item.item|replace('eth',prefix) }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Edit DEVICE in new network-script files
             lineinfile:
               regexp: "^DEVICE=.*"
               line: "DEVICE={{ item.item|replace('eth',prefix) }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Backup old eth network-script files
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path }}.bak"
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
           - name: Remove old eth network-script files
             file:
               path: "{{ item.stat.path }}"
               state: absent
             with_items: "{{ nic_result.results }}"
             when: item.stat.exists
       - name: Rename route files
         block:
           - name: Check that route files exists
             stat:
               path: /etc/sysconfig/network-scripts/route-{{ item }}
             register: route_result
             with_items: "{{ eth_interfaces }}"
           - name: Copy route files using the new prefix
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Update prefix in route files that use IP command arguments format
             replace:
               regexp: "eth"
               replace: "{{ prefix }}"
               path: "{{ item.stat.path|replace('eth',prefix) }}"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Backup old route files
             copy:
               remote_src: True
               src: "{{ item.stat.path }}"
               dest: "{{ item.stat.path }}.bak"
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
           - name: Remove old route files
             file:
               path: "{{ item.stat.path }}"
               state: absent
             with_items: "{{ route_result.results }}"
             when: item.stat.exists
       - name: Perform a final regex for any remaining eth prefixes in ifcfg files
         block:
           - name: Get a list of all ifcfg files
             find:
               paths: /etc/sysconfig/network-scripts/
               patterns: 'ifcfg-*'
             register: ifcfg_files
           - name: Perform final regex on ifcfg files
             replace:
               path: "{{ item[0].path }}"
               regexp: "{{ item[1] }}"
               replace: "{{ item[1]|replace('eth',prefix) }}"
             with_nested:
               - "{{ ifcfg_files.files }}"
               - "{{ eth_interfaces }}"

   Note

   You will use this playbook to rename the overcloud NICs at a later stage in the upgrade process.

3. Run the playbook-nics.yaml playbook on the undercloud:

   $ ansible-playbook -c local -i localhost, playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -c local -i localhost, -e prefix="mynic" ~/playbook-nics.yaml

4. Reboot the undercloud node using the standard reboot procedures. For more information, see "Rebooting nodes".

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

   $ sudo grep PermitRootLogin /etc/ssh/sshd_config

3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

   PermitRootLogin no

4. Save the file.

Procedure

1. Check the current list of hardware types enabled:

   $ source ~/stackrc
   $ openstack baremetal driver list --type dynamic

2. If you use a hardware type driver that is not enabled, enable the driver using the enabled_hardware_types parameter in the undercloud.conf file:

   enabled_hardware_types = ipmi,redfish,idrac

3. Save the file and refresh the undercloud:

   $ openstack undercloud install

4. Run the following commands, substituting the OLDDRIVER and NEWDRIVER variables for your power management type:

   $ source ~/stackrc
   $ OLDDRIVER="pxe_ipmitool"
   $ NEWDRIVER="ipmi"
   $ for NODE in $(openstack baremetal node list --driver $OLDDRIVER -c UUID -f value) ; do openstack baremetal node set $NODE --driver $NEWDRIVER; done

Procedure

1. Log in to the undercloud as the stack user.

2. Disable the main OpenStack services on the undercloud:

   $ sudo systemctl stop openstack-* httpd haproxy mariadb rabbitmq* docker xinetd

3. Remove the main OpenStack services from the undercloud, except OpenvSwitch and certain Python 2 packages that are required for the upgrade:

   $ sudo yum -y remove *el7ost* galera* haproxy* \
       httpd mysql* pacemaker* xinetd python-jsonpointer \
       qemu-kvm-common-rhev qemu-img-rhev rabbit* \
       redis* \
       -- \
       -*openvswitch* -python-docker -python-PyMySQL \
       -python-pysocks -python2-asn1crypto -python2-babel \
       -python2-cffi -python2-cryptography -python2-dateutil \
       -python2-idna -python2-ipaddress -python2-jinja2 \
       -python2-jsonpatch -python2-markupsafe -python2-pyOpenSSL \
       -python2-requests -python2-six -python2-urllib3 \
       -python-httplib2 -python-passlib -python2-netaddr -ceph-ansible

4. Remove the content from the /etc/httpd and /var/lib/docker directories:

   $ sudo rm -rf /etc/httpd /var/lib/docker

Procedure

1. Log in to the undercloud as the stack user.

2. Install the Leapp utility:

   $ sudo yum install leapp

3. Download the additional required data files (RPM package changes and RPM repository mapping) attached to the Knowledge Base article "Data required by the Leapp utility for an in-place upgrade from RHEL 7 to RHEL 8" and place these files in the /etc/leapp/files/ directory.

4. Update your Red Hat subscription:

   • If your undercloud uses the Red Hat Customer Portal for registration, refresh your current subscription to obtain access to the Red Hat Enterprise Linux 8.2 content:

     $ sudo subscription-manager refresh

   • If your undercloud uses a Red Hat Satellite server for registration, re-register the undercloud to a content view associated with your Red Hat OpenStack Platform 16.1 activation key.

     $ sudo subscription-manager register --force --org ORG --activationkey ACTIVATION_KEY --release 8.2

     Note

     The content view that you create for Red Hat OpenStack Platform 16.1 should contain content for Red Hat Enterprise Linux 8.2.

5. Red Hat OpenStack Platform 16.1 uses a newer version of Open vSwitch`. Substitute the Open vSwitch version through the to_remove and to_install transaction files:

   $ echo 'openvswitch2.11' | sudo tee -a /etc/leapp/transaction/to_remove
   $ echo 'openvswitch2.13' | sudo tee -a /etc/leapp/transaction/to_install

6. Retain the Red Hat Ceph Storage 3 version of ceph-ansible through the upgrade with the to_keep transaction file:

   $ echo 'ceph-ansible' | sudo tee -a /etc/leapp/transaction/to_keep

7. Start the Leapp upgrade process:

   $ sudo leapp upgrade --debug --enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms --enablerepo ansible-2.9-for-rhel-8-x86_64-rpms

   Use the --enablerepo option to set the repositories that you want to enable during the Leapp upgrade process. You must include these repositories to facilitate the Red Hat OpenStack Platform 16.1 transition, especially with the newer version of Open vSwitch.

8. Wait for the leapp upgrade command to successfully complete.

9. Create an empty .autorelabel file in your root directory:

   $ sudo touch /.autorelabel

   After a reboot, SELinux detects this file and automatically relabels the file system.

10. Reboot the undercloud:

    $ sudo reboot

Procedure

1. Log in to the undercloud as the stack user.

2. Lock the undercloud to a specific verison with the subscription-manager release command:

   $ sudo subscription-manager release --set=8.2

Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms --enable=advanced-virt-for-rhel-8-x86_64-rpms

   These repositories contain packages that the director installation requires.

3. Set the container-tools repository module to version 2.0:

   [stack@director ~]$ sudo dnf module disable -y container-tools:rhel8
   [stack@director ~]$ sudo dnf module enable -y container-tools:2.0

4. Set the virt repository module to version 8.2:

   [stack@director ~]$ sudo dnf module disable -y virt:rhel
   [stack@director ~]$ sudo dnf module enable -y virt:8.2

5. Synchronize the operating system to ensure that your system packages match the operating system version:

   [stack@director ~]$ sudo dnf distro-sync -y
   [stack@director ~]$ sudo reboot

Procedure

1. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. With this option, director pulls the necessary images from the Red Hat Container Catalog and pushes the images to the registry on the undercloud. Director uses the undercloud registry as the container image source. To pull container images directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file specifies an environment file that includes include the parameters for preparing your container images. In this example, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Modify the containers-prepare-parameter.yaml to suit your requirements.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the containers-prepare-parameter.yaml file.

3. Add the transitional container parameters to set in the ContainerImagePrepare parameter. The parameters you set differ based on whether you use a director-deployed Ceph Storage cluster or and external Ceph Storage cluster.

   • If your deployment uses a Red Hat Ceph Storage cluster that was deployed using director, add the following parameters:

     parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
         set:
           ...
           name_prefix_stein: openstack-
           name_suffix_stein: ''
           namespace_stein: registry.redhat.io/rhosp15-rhel8
           tag_stein: 15.0
           ceph3_namespace: registry.redhat.io/rhceph
           ceph3_tag: latest
           ceph3_image: rhceph-3-rhel7
           ...

     • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
     • The ceph3_* parameters define the current Red Hat Ceph Storage container images that the overcloud uses. The overcloud requires both the ceph3_* and ceph_* parameters for the transition from Red Hat Ceph Storage 3 to 4.
     • If you use Red Hat Satellite for container image storage, set the namespaces to the image locations on the Red Hat Satellite server.

   • If you are upgrading with external Ceph deployments, add the following parameters:

     parameter_defaults:
       ContainerImagePrepare:
       - push_destination: true
         set:
           ...
           name_prefix_stein: openstack-
           name_suffix_stein: ''
           namespace_stein: registry.redhat.io/rhosp15-rhel8
           tag_stein: 15.0
           ceph_namespace: registry.redhat.io/rhceph
           ceph_tag: latest
           ceph_image: rhceph-4-rhel8
           ...

     • The *_stein parameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
     • The ceph_* parameters define the current Red Hat Ceph Storage 4 container images that the overcloud uses.
     • If you use Red Hat Satellite for container image storage, set the namespaces to the image locations on the Red Hat Satellite server.

4. Change the neutron_driver parameter to openvswitch:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       set:
         ...
         neutron_driver: openvswitch
         ...

   The upgrade retains Open vSwitch compatibility throughout the process. After you complete the upgrade to Red Hat OpenStack Platform 16.1, you migrate the overcloud from Open vSwitch to Open Virtual Network (OVN).

5. Save the containers-prepare-parameter.yaml file.

Procedure

1. Log in to your undercloud host as the stack user.
2. Edit the undercloud.conf file.

3. Add the following parameter to the DEFAULT section in the file:

   container_images_file = /home/stack/containers-prepare-parameter.yaml

   This parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.

4. Check the generate_service_certificate parameter. The default for this parameter changes from false to true, which enables SSL/TLS on your undercloud, during the upgrade.
5. Check the local_interface parameter if you have migrated to a predictable NIC naming convention.
6. If you set the masquerade_network parameter in Red Hat OpenStack Platform 13, remove this parameter and set masquerade = true for each subnet.
7. Check all other parameters in the file for any changes.
8. Save the file.

Procedure

1. Run the following command to upgrade the director on the undercloud:

   $ openstack undercloud upgrade

   This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

   Note

   The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the -y option:

   $ openstack undercloud upgrade -y

2. The script also starts all OpenStack Platform service containers on the undercloud automatically. You manage each service through a systemd resource. Check the systemd resources:

   $ sudo systemctl list-units "tripleo_*"

   Each systemd service controls a container. Check the enabled containers using the following command:

   $ sudo podman ps

3. The script adds the stack user to the docker group to ensure that the stack user has access to container management commands. Refresh the stack user permissions with the following command:

   $ exec su -l stack

   The command prompts you to log in again. Enter the stack user password.

4. To initialize the stack user to use the command line tools, run the following command:

   $ source ~/stackrc

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

   (undercloud) $

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group pre-upgrade option:

   $ openstack tripleo validator run --group pre-upgrade

3. Review the results of the validation report. To view detailed output from a specific validation, run the openstack tripleo validator show run command against the UUID of the specific validation from the report:

   $ openstack tripleo validator show run <UUID>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to disable fencing:

   $ ssh heat-admin@CONTROLLER_IP "sudo pcs property set stonith-enabled=false"

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Create a static inventory file of all nodes:

   $ tripleo-ansible-inventory --static-yaml-inventory ~/inventory.yaml --stack STACK_NAME

   If you are not using the default overcloud stack name, set your stack name with the --stack STACK NAME option replacing STACK NAME with the name of your stack.

4. To execute Ansible playbooks on your environment, run the ansible-playbook command and include the full path of the dynamic inventory tool using the -i option. For example:

   (undercloud) $ ansible-playbook -i ~/inventory.yaml PLAYBOOK

Procedure

1. Log in to the undercloud as the stack user.

2. Create an environment file called upgrades-environment.yaml in your templates directory:

   $ touch templates/upgrades-environment.yaml

3. Edit the file and add the following mandatory content:

   parameter_defaults:
     UpgradeInitCommand: |
       sudo dnf module disable -y container-tools:rhel8
       sudo dnf module enable -y container-tools:2.0
       sudo dnf module disable -y virt:rhel
       sudo dnf module enable -y virt:8.2
     UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"

   • UpgradeInitCommand sets a series of bash commands to run that director runs before starting the upgrade on a node. In this context, set this parameter to use the dnf commands to switch from the default container-tools:rhel8 module to container-tools:2.0.
   • UpgradeLeappCommandOptions sets additional options to pass to the Leapp tool. In this context, set this parameter to enable Red Hat OpenStack Platform repositories to help with the Leapp transition.
4. Save the upgrades-environment.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/leapp/files directory on the undercloud:

   $ sudo ls /etc/leapp/files/

Procedure

1. Log in to the undercloud as the stack user.

2. Run the playbook-nics.yaml playbook on all overcloud nodes:

   $ ansible-playbook -i ~/inventory.yaml playbook-nics.yaml

   The playbook sets the new NIC prefix to em. To set a different NIC prefix, set the prefix variable when running the playbook:

   $ ansible-playbook -i ~/inventory.yaml -e prefix="mynic" playbook-nics.yaml

3. Reboot overcloud nodes using the standard reboot procedures. For more information, see "Rebooting nodes".

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-ssh.yaml and paste the following content in the file:

   ---
   - name: Configure SSH PermitRootLogin parameter
     hosts: overcloud
     become: yes
     tasks:
       - name: Set the PermitRootLogin parameter to no
         lineinfile:
           path: /etc/ssh/sshd_config
           state: present
           line: "PermitRootLogin no"

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-ssh.yaml

Procedure

1. Search for custom environment files that use composable services. You usually store custom environment files in the /home/stack/templates directory:

   $ cd ~/templates/
   $ grep "OS::TripleO::Services" *

   In this scenario, one of the files shows an outdated mapping:

     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-mgr.yaml

2. Identify the new ceph-mgr.yaml location in /usr/share/openstack-tripleo-heat-templates/. This file is now located in the `deployment/ceph-ansible' directory:

   $ find /usr/share/openstack-tripleo-heat-templates/ -name ceph-mgr.yaml
   /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

3. Edit the service in the custom environment file:

   resource_registry:
     OS::TripleO::Services::CephMgr: /usr/share/openstack-tripleo-heat-templates/deployment/ceph-ansible/ceph-mgr.yaml

   Save the file.

Procedure

1. Log in to the undercloud as the stack user.

2. Obtain the control plane host name on the undercloud:

   $ sudo hiera container_image_prepare_node_names
   ["undercloud.ctlplane.localdomain"]

3. Edit the containers-prepare-parameter.yaml file and add the DockerInsecureRegistryAddress parameter with a YAML list that contains the undercloud control plane host name:

   parameter_defaults:
     DockerInsecureRegistryAddress:
     - undercloud.ctlplane.localdomain:8787
     ...

   You must also append the port number of the overcloud registry to the host name value. The port number is 8787.

Procedure

1. Log in to the undercloud as the stack user.

2. Set the Compute naming format:

   • If you use a custom roles_data file, edit the custom roles_data file and set the HostnameFormatDefault parameter for the Compute role:

     - name: Compute
       …​
       HostnameFormatDefault: '%stackname%-compute-%index%'
       …​

     Save the custom roles_data file.

   • If you use the default roles_data file in openstack-tripleo-heat-templates, set the naming format in an environment file. Edit the environment file with your node counts and flavors, which is usually named node-info.yaml. Add the ComputeHostnameFormat parameter to the parameter_defaults section:

     parameter_defaults:
       …​
       ComputeHostnameFormat: '%stackname%-compute-%index%'
       …​

     Save the node-info.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Edit your custom overcloud SSL/TLS public endpoint file, which is usually named ~/templates/enable-tls.yaml.

4. Remove the NodeTLSData resource from the `resource_registry:

   resource_registry:
     OS::TripleO::NodeTLSData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/tls-cert-inject.yaml
     …​

   The overcloud deployment uses a new service in HAProxy to determine if SSL/TLS is enabled.

   Note

   If this is the only resource in the resource_registry section of the enable-tls.yaml file, remove the complete resource_registry section.

5. Save the SSL/TLS public endpoint file file.

Procedure

1. Edit the post-configuration template.

2. Under the parameters section, add the EndpointMap parameter and its sub-parameters:

   parameters:
     servers:
       type: json
     nameserver_ip:
       type: string
     DeployIdentifier:
       type: string
     EndpointMap:
       default: {}
       type: json

3. Save the template.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file (templates/rhsm.yaml) to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_repos:
         - rhel-8-for-x86_64-baseos-eus-rpms
         - rhel-8-for-x86_64-appstream-eus-rpms
         - rhel-8-for-x86_64-highavailability-eus-rpms
         …​
       rhsm_username: "myusername"
       rhsm_password: "p@55w0rd!"
       rhsm_org_id: "1234567"
       rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
       rhsm_method: "portal"
       rhsm_release: 8.2

   • The resource_registry section associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.
   • The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.
3. Save the environment file.

Procedure

1. Create an environment file (templates/rhsm.yaml) to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     ControllerParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-eus-rpms
           - rhel-8-for-x86_64-appstream-eus-rpms
           - rhel-8-for-x86_64-highavailability-eus-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - advanced-virt-for-rhel-8-x86_64-rpms
           - openstack-16.1-for-rhel-8-x86_64-rpms
           - rhceph-4-mon-for-rhel-8-x86_64-rpms
           - rhceph-4-tools-for-rhel-8-x86_64-rpms
           - fast-datapath-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "55d251f1490556f3e75aa37e89e10ce5"
         rhsm_method: "portal"
         rhsm_release: 8.2
     ComputeParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-eus-rpms
           - rhel-8-for-x86_64-appstream-eus-rpms
           - rhel-8-for-x86_64-highavailability-eus-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - advanced-virt-for-rhel-8-x86_64-rpms
           - openstack-16.1-for-rhel-8-x86_64-rpms
           - rhceph-4-tools-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "55d251f1490556f3e75aa37e89e10ce5"
         rhsm_method: "portal"
         rhsm_release: 8.2
     CephStorageParameters:
       RhsmVars:
         rhsm_repos:
           - rhel-8-for-x86_64-baseos-rpms
           - rhel-8-for-x86_64-appstream-rpms
           - rhel-8-for-x86_64-highavailability-rpms
           - ansible-2.9-for-rhel-8-x86_64-rpms
           - openstack-16.1-deployment-tools-for-rhel-8-x86_64-rpms
           - rhceph-4-osd-for-rhel-8-x86_64-rpms
           - rhceph-4-tools-for-rhel-8-x86_64-rpms
         rhsm_username: "myusername"
         rhsm_password: "p@55w0rd!"
         rhsm_org_id: "1234567"
         rhsm_pool_ids: "68790a7aa2dc9dc50a9bc39fabc55e0d"
         rhsm_method: "portal"
         rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The ControllerParameters, ComputeParameters, and CephStorageParameters use their own RhsmVars parameter to pass subscription details to their respective roles.

   Note

   Set the RhsmVars parameter within the CephStorageParameters parameter to use a Red Hat Ceph Storage subscription and repositories specific to Ceph Storage. Ensure the rhsm_repos parameter contains the standard Red Hat Enterprise Linux repositories instead of the Extended Update Support (EUS) repositories that Controller and Compute nodes require.

3. Save the environment file.

Procedure

1. Exclude the rhel-registration environment files from future deployments operations. In most cases, exclude the following files:

   • rhel-registration/environment-rhel-registration.yaml
   • rhel-registration/rhel-registration-resource-registry.yaml

2. If you use a custom roles_data file, ensure that each role in your roles_data file contains the OS::TripleO::Services::Rhsm composable service. For example:

   - name: Controller
     description: |
       Controller role that has all the controller services loaded and handles
       Database, Messaging and Network functions.
     CountDefault: 1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Rhsm
       ...

3. Add the environment file for rhsm composable service parameters to future deployment operations.

Procedure

1. Create an environment file (templates/rhsm.yaml) to store the configuration.

2. Include your configuration in the environment file. For example:

   resource_registry:
     OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/deployment/rhsm/rhsm-baremetal-ansible.yaml
   parameter_defaults:
     RhsmVars:
       rhsm_activation_key: "myactivationkey"
       rhsm_method: "satellite"
       rhsm_org_id: "ACME"
       rhsm_server_hostname: "satellite.example.com"
       rhsm_baseurl: "https://satellite.example.com/pulp/repos"
       rhsm_release: 8.2

   The resource_registry associates the rhsm composable service with the OS::TripleO::Services::Rhsm resource, which is available on each role.

   The RhsmVars variable passes parameters to Ansible for configuring your Red Hat registration.

3. Save the environment file.

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file called playbook-satellite.yaml and paste the following content in the file:

   - name: Pre-install leapp
     hosts: overcloud
     become: yes
     tasks:
       - name: Pre-install leapp
         yum:
           name:
             - leapp
             - leapp-repository
           state: installed
       - name: Remove katello-host-tools-fact-plugin
         yum:
           name:
             - katello-host-tools-fact-plugin
           state: removed
       - name: Register system
         import_role:
           name: redhat-subscription
         vars:
           rhsm_activation_key: "osp16-key"
           rhsm_method: "satellite"
           rhsm_org_id: "ACME"
           rhsm_server_hostname: "satellite.example.com"
           rhsm_baseurl: "https://satellite.example.com/pulp/repos"
           rhsm_release: "8.2"
           rhsm_force_register: True

   Modify the rhsm_* variables to suit your Satellite server.

3. Run the playbook:

   $ ansible-playbook -i ~/inventory.yaml playbook-satellite.yaml

1. Upgrade the undercloud but retain version 3 of ceph-ansible
2. Start the overcloud upgrade

3. Perform the following tasks for each node that hosts Ceph Storage containerized services:

   1. Migrate the Ceph Storage containerized services to Podman
   2. Upgrade the operating system
   3. Upgrade the OpenStack Platform services, which relaunches Ceph Storage version 3 containerized services
4. Complete the overcloud upgrade
5. Upgrade ceph-ansible to version 4 on the undercloud
6. Upgrade to Red Hat Ceph Storage 4 on the overcloud

Procedure

1. Log in to the undercloud as the stack user.

2. Run the dnf command to check the version of the ceph-ansible package:

   $ sudo dnf info ceph-ansible

   The command output shows version 3 of the ceph-ansible package:

   Installed Packages
   Name         : ceph-ansible
   Version      : 3.xx.xx.xx
   ...

Procedure

1. Log in to the undercloud as the stack user.

2. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named ceph-config.yaml and you can find it in your templates directory:

   $ vi /home/stack/templates/ceph-config.yaml

3. Add the CephAnsibleRepo parameter to the parameter_defaults section:

   parameter_defaults:
     ...
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
     ...

   CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.

4. Save the ceph-config.yaml file.

Procedure

1. Log in to the node that is running the ceph-mon service. This node is usually a Controller node or a standalone Ceph Monitor node.

2. Enter the following command to view the status of the Ceph cluster:

   $ docker exec ceph-mon-$HOSTNAME ceph -s

3. Confirm that the health status of the cluster is HEALTH_OK and that all of the OSDs are up.

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to the undercloud as the stack user.

2. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named ceph-config.yaml and you can find it in your templates directory:

   $ vi /home/stack/templates/ceph-config.yaml

3. Add the CephAnsibleRepo parameter to the parameter_defaults section:

   parameter_defaults:
     ...
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
     ...

   CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.

4. Save the ceph-config.yaml file.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. On the undercloud, create a file called update-nic-templates.sh and include the following content in the file:

   #!/bin/bash
   STACK_NAME="overcloud"
   ROLES_DATA="/usr/share/openstack-tripleo-heat-templates/roles_data.yaml"
   NETWORK_DATA="/usr/share/openstack-tripleo-heat-templates/network_data.yaml"
   NIC_CONFIG_LINES=$(openstack stack environment show $STACK_NAME | grep "::Net::SoftwareConfig" | sed -E 's/ *OS::TripleO::// ; s/::Net::SoftwareConfig:// ; s/ http.*user-files/ /')
   echo "$NIC_CONFIG_LINES" | while read LINE; do
       ROLE=$(echo "$LINE" | awk '{print $1;}')
       NIC_CONFIG=$(echo "$LINE" | awk '{print $2;}')
   
       if [ -f "$NIC_CONFIG" ]; then
           echo "Updating template for $ROLE role."
           python3 /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py \
               --tht-dir /usr/share/openstack-tripleo-heat-templates \
               --roles-data $ROLES_DATA \
               --network-data $NETWORK_DATA \
               --role-name "$ROLE" \
               --discard-comments yes \
               --template "$NIC_CONFIG"
       else
           echo "No NIC template detected for $ROLE role. Skipping $ROLE role."
       fi
   done

   • If you use a custom overcloud name, Set the STACK_NAME variable to the name of your overcloud. The default name for an overcloud stack is overcloud.
   • If you use a custom roles_data file, set the ROLES_DATA variable to the location of the custom file. If you use the default roles_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
   • If you use a custom network_data file, set the NETWORK_DATA variable to the location of the custom file. If you use the default network_data file, leave the variable as /usr/share/openstack-tripleo-heat-templates/network_data.yaml
   • Run /usr/share/openstack-tripleo-heat-templates/tools/merge-new-params-nic-config-script.py -h to see a list of options to add to the script.

4. Add executable permissions to the script:

   $ chmod +x update-nic-templates.sh

5. Run the script:

   $ ./update-nic-templates.sh

   The script saves a copy of each custom NIC template and updates each template with the missing parameters. The script also skips any roles that do not have a custom template:

   No NIC template detected for BlockStorage role. Skipping BlockStorage role.
   Updating template for CephStorage role.
   The original template was saved as: /home/stack/templates/custom-nics/ceph-storage.yaml.20200903144835
   The update template was saved as: /home/stack/templates/custom-nics/ceph-storage.yaml
   Updating template for Compute role.
   The original template was saved as: /home/stack/templates/custom-nics/compute.yaml.20200903144838
   The update template was saved as: /home/stack/templates/custom-nics/compute.yaml
   Updating template for Controller role.
   The original template was saved as: /home/stack/templates/custom-nics/controller.yaml.20200903144841
   The update template was saved as: /home/stack/templates/custom-nics/controller.yaml
   No NIC template detected for ObjectStorage role. Skipping ObjectStorage role.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on a Controller node:

   $ sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name

   To run this command from the undercloud, run the following SSH command and substitute CONTROLLER_IP with the IP address of any Controller node in your environment:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

   This example uses overcloud-controller-0 as the value of the bootstrap node. Substitute this value for your own bootstrap node.

3. Upgrade the bootstrap Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

4. Upgrade the next Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Log in to a node with Ceph MON services:

3. Disable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd set noout
   $ sudo podman exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

4. Log out of the node with Ceph MON services and return to the undercloud.

5. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

6. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

7. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.

9. Enable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd unset noout
   $ sudo podman exec -it CONTAINER ceph osd unset norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

10. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on a Controller node:

   $ sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name

   To run this command from the undercloud, run the following SSH command and substitute CONTROLLER_IP with the IP address of any Controller node in your environment:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

   This example uses overcloud-controller-0 as the value of the bootstrap node. Substitute this value for your own bootstrap node.

3. Upgrade the bootstrap Controller node:

   1. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   2. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   3. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   4. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

4. Upgrade the next Controller node:

   1. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   2. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   2. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap node by running the following command on a Controller-based node:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

3. Upgrade the bootstrap node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

4. Upgrade each Pacemaker-based node:

   1. If the node any contains Ceph Storage containers, run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-database-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-database-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-database-0

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include any previously upgraded node in the --limit option.

5. Repeat the upgrade process on each Pacemaker-based node until you have upgraded all Pacemaker-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-networker-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

3. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-networker-0

   This command performs the Red Hat OpenStack Platform upgrade.

4. Repeat the upgrade process on each node until you have upgraded all Controller-based node.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Select a Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-0

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

3. Select the next Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-1

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

4. Select the final Ceph MON node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephmon-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephmon-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephmon-2

      This command runs the config-download playbooks and configures the composable services on the Ceph MON node. This step does not upgrade the Ceph MON nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Log in to a node with Ceph MON services:

3. Disable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd set noout
   $ sudo podman exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

4. Log out of the node with Ceph MON services and return to the undercloud.

5. Select a Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-0

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

6. Select the next Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-1

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

7. Select the final Ceph Storage node and upgrade the operating system:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-cephstorage-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-cephstorage-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-cephstorage-2

      This command runs the config-download playbooks and configures the composable services on the Ceph Storage node. This step does not upgrade the Ceph Storage nodes to Red Hat Ceph Storage 4. The Red Hat Ceph Storage 4 upgrade occurs in a later procedure.

8. After you upgrade all HCI nodes, log into a node that hosts Ceph MON services.

9. Enable OSD exclusion and rebalancing:

   $ sudo podman ps | grep ceph-mon
   $ sudo podman exec -it CONTAINER ceph osd unset noout
   $ sudo podman exec -it CONTAINER ceph osd unset norebalance

   Replace CONTAINER with the name of the container running Ceph MON.

10. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"

3. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

4. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-compute-0

   This command performs the Red Hat OpenStack Platform upgrade.

5. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-compute-0,overcloud-compute-1,overcloud-compute-2

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the upgrade preparation command:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
3. Wait until the upgrade preparation completes.

4. Download the container images:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags container_image_prepare

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the bootstrap Controller node by running the following command on a Controller node:

   $ sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name

   To run this command from the undercloud, run the following SSH command and substitute CONTROLLER_IP with the IP address of any Controller node in your environment:

   $ ssh heat-admin@CONTROLLER_IP "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

   This example uses overcloud-controller-0 as the value of the bootstrap node. Substitute this value for your own bootstrap node.

3. Upgrade the bootstrap Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-0

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.

      • Performs a reboot as a part of the Leapp upgrade.

        Important

        The next command causes an outage on the control plane. You cannot perform any standard operations on the overcloud during the next few steps.

   3. Run the external upgrade command with the system_upgrade_transfer_data tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags system_upgrade_transfer_data

      This command copies the latest version of the database from an existing node to the bootstrap node.

   4. Run the upgrade command with the nova_hybrid_state tag and run only the upgrade_steps_playbook.yaml playbook:

      $ openstack overcloud upgrade run --stack STACK NAME --playbook upgrade_steps_playbook.yaml --tags nova_hybrid_state --limit all

      This command launches temporary 16.1 containers on Compute nodes to help facilitate workload migration when you upgrade Compute nodes at a later step.

   5. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0

      This command performs the Red Hat OpenStack Platform upgrade.

      Important

      The control plane becomes active when this command finishes. You can perform standard operations on the overcloud again.

4. Upgrade the next Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-1

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag on the next Controller node:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-1

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1

      This command performs the Red Hat OpenStack Platform upgrade. In addition to this node, include the previously upgraded bootstrap node in the --limit option.

5. Upgrade the final Controller node:

   1. Run the external upgrade command with the ceph_systemd tag:

      $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-2

      This command performs the following functions:

      • Changes the systemd units that control the Ceph Storage containers to use Podman management.
      • Limits actions to the selected Controller node using the ceph_ansible_limit variable.

      This step is a preliminary measure to prepare the Ceph Storage services for The leapp upgrade.

   2. Run the upgrade command with the system_upgrade tag:

      $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-controller-2

      This command performs the following actions:

      • Performs a Leapp upgrade of the operating system.
      • Performs a reboot as a part of the Leapp upgrade.

   3. Run the upgrade command with no tags:

      $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

      This command performs the Red Hat OpenStack Platform upgrade. Include all Controller nodes in the --limit option.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Migrate your instances. For more information on migration strategies, see "Migrating virtual machines between Compute nodes"
3. Log in to a node with Ceph MON services:

4. Disable OSD exclusion and rebalancing:

   $ sudo docker ps | grep ceph-mon
   $ sudo docker exec -it CONTAINER ceph osd set noout
   $ sudo docker exec -it CONTAINER ceph osd set norebalance

   Replace CONTAINER with the name of the container that runs Ceph MON.

5. Log out of the node with Ceph MON services and return to the undercloud.

6. Run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

   This command performs the following functions:

   • Changes the systemd units that control the Ceph Storage containers to use Podman management.
   • Limits actions to the selected Ceph Storage node using the ceph_ansible_limit variable.

   This step is a preliminary measure to prepare the Ceph Storage services for the leapp upgrade.

7. Run the upgrade command with the system_upgrade tag:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0

   This command performs the following actions:

   • Performs a Leapp upgrade of the operating system.
   • Performs a reboot as a part of the Leapp upgrade.

8. Run the upgrade command with no tags:

   $ openstack overcloud upgrade run --stack STACK NAME --limit overcloud-computehci-0

   This command performs the Red Hat OpenStack Platform upgrade.

9. To upgrade multiple Compute nodes in parallel, set the --limit option to a comma-separated list of nodes that you want to upgrade. First run the external upgrade command with the ceph_systemd tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph_systemd -e ceph_ansible_limit=overcloud-computehci-0

   Then perform the system_upgrade task:

   $ openstack overcloud upgrade run --stack STACK NAME --tags system_upgrade --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2

   Then perform the standard OpenStack service upgrade:

   $ openstack overcloud upgrade run --stack STACK NAME  --limit overcloud-computehci-0,overcloud-computehci-1,overcloud-computehci-2

10. After you have upgraded all HCI nodes, log in to a node with Ceph MON services.

11. Enable OSD exclusion and rebalancing:

    $ sudo podman ps | grep ceph-mon
    $ sudo podman exec -it CONTAINER ceph osd unset noout
    $ sudo podman exec -it CONTAINER ceph osd unset norebalance

    Replace CONTAINER with the name of the container that runs Ceph MON.

12. Log out of the node with Ceph MON services and return to the undercloud.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Edit the containers-prepare-parameter.yaml file and remove the following parameters and their values:

   • ceph3_namespace
   • ceph3_tag
   • ceph3_image
   • name_prefix_stein
   • name_suffix_stein
   • namespace_stein
   • tag_stein

3. Run the upgrade finalization command:

   $ openstack overcloud upgrade converge \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       -e /home/stack/templates/rhsm.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
       …​

   Include the following options relevant to your environment:

   • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
   • The environment file (rhsm.yaml) with the registration and subscription parameters (-e).
   • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
   • The environment file (neutron-ovs.yaml) to maintain OVS compatibility.
   • Any custom configuration environment files (-e) relevant to your deployment.
   • If applicable, your custom roles (roles_data) file using --roles-file.
   • If applicable, your composable network (network_data) file using --networks-file.
   • If you use a custom stack name, pass the name with the --stack option.
4. Wait until the stack synchronization completes.

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run --stack STACK NAME --tags ceph

3. Wait until the Ceph Storage upgrade completes.

Procedure

1. Log in as the heat-admin user on a node with Ceph MON containers, such as Controller nodes or standalone Ceph MON nodes. For example, in a standard overcloud deployment, overcloud-controller-1 uses Ceph MON containers.

2. Query the Ceph cluster to see what driver is in use by the OSDs:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]#

3. If any line returns "objectstore": "filestore", that node requires OSD migration.

Procedure

1. Ensure that osd_objectstore in the CephAnsibleDisksConfig parameter does not default to filestore. If the osd_objectstore parameter is present in any of your custom heat environment files, you must define the value bluestore explicitly or remove it:

   parameter_defaults:
     CephAnsibleDisksConfig:
       devices:
         - /dev/sdb
         - /dev/sdc
         - /dev/sdd
       osd_scenario: lvm
      osd_objectstore: bluestore

   Note

   If you have any specific FileStore configuration with, for example, journals, ensure that you update the configuration accordingly. For more information about advanced configurations, see Mapping the Ceph Storage Node Disk Layout in the Deploying an overcloud with containerized Red Hat Ceph guide.

2. Log in to the undercloud as the stack user.

3. Enter the openstack overcloud external-upgrade run command with the ceph_fstobs tag. Replace <NODE_NAME> with the name of the Ceph OSD node you want to upgrade. You can use the openstack server list command to find the node name.

   [stack@undercloud ~] $ openstack overcloud external-upgrade run --tags ceph_fstobs -e ceph_ansible_limit=<NODE_NAME> | tee oc-fstobs.log

4. Log in to a node that has Ceph MON services and query the Ceph cluster to check the status of the OSDs of the node to ensure it is migrated before you continue to the next OSD:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c '.[] | select(.hostname == "<NODE_NAME>") | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]# exit

   Replace <NODE_NAME> with the name of the node that was migrated. If the result shows that the OSDs use BlueStore, its migration is successful.

5. To view additional details about a specific OSD, enter the following command:

   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph osd metadata <ID>"

   Replace <ID> with the ID of the OSD you want to query.

6. Wait for the cluster to synchronize before you start the migration on the next node.

   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c  "ceph  -s"

7. Verify the output of the command above and ensure that the health of the cluster is HEALTH_OK and the PGs are in the active+clean state.
8. Repeat the migration process for each node in the storage cluster.

Procedure

1. Log in as the heat-admin user to a ceph-mon container that is hosted on one of the Controller nodes.

2. Query the Ceph cluster:

   [heat-admin@overcloud-controller-1 ~]$ sudo -i
   [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
   [root@overcloud-controller-1 ~]#

Procedure

1. Remove the unnecessary packages

   $ sudo dnf -y remove --exclude=python-pycadf-common python2*

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group post-upgrade option:

   $ openstack tripleo validator run --group post-upgrade

3. Review the results of the validation report. To view detailed output from a specific validation, run the openstack tripleo validator show run command against the UUID of the specific validation from the report:

   $ openstack tripleo validator show run <UUID>

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Install the packages containing the overcloud QCOW2 archives:

   $ sudo dnf install rhosp-director-images rhosp-director-images-ipa

4. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

5. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done
   $ cd ~

6. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

7. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

Procedure

1. Log in to the undercloud as the stack user.

2. If your Compute nodes support simultaneous multithreading (SMT) but you created instances with the hw:cpu_thread_policy=isolate policy, you must perform one of the following options:

   • Unset the hw:cpu_thread_policy thread policy and resize the instances:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Unset the hw:cpu_thread_policy property of the flavor:

        (overcloud) $ openstack flavor unset --property hw:cpu_thread_policy <flavor>

        Note

        • Unsetting the hw:cpu_thread_policy attribute sets the policy to the default prefer policy, which sets the instance to use an SMT-enabled Compute node if available. You can also set the hw:cpu_thread_policy attribute to require, which sets a hard requirements for an SMT-enabled Compute node.
        • If the Compute node does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling will fail. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The default prefer policy ensures that thread siblings are used when available.
        • If you use hw:cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

     3. Convert the instances to use the new thread policy.

        (overcloud) $ openstack server resize --flavor <flavor> <server>
        (overcloud) $ openstack server resize confirm <server>

        Repeat this step for all pinned instances using the hw:cpu_thread_policy=isolated policy.

   • Migrate instances from the Compute node and disable SMT on the Compute node:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Disable the Compute node from accepting new virtual machines:

        (overcloud) $ openstack compute service list
        (overcloud) $ openstack compute service set <hostname> nova-compute --disable

     3. Migrate all instances from the Compute node. For more information on instance migration, see "Migrating virtual machine instances between Compute nodes".
     4. Reboot the Compute node and disable SMT in the BIOS of the Compute node.
     5. Boot the Compute node.

     6. Re-enable the Compute node:

        (overcloud) $ openstack compute service set <hostname> nova-compute --enable

3. Source the stackrc file:

   $ source ~/stackrc

4. Edit the environment file that contains the NovaVcpuPinSet parameter.

5. Migrate the CPU pinning configuration from the NovaVcpuPinSet parameter to NovaComputeCpuDedicatedSet and NovaComputeCpuSharedSet:

   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuDedicatedSet for hosts that were previously used for pinned instances.
   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuSharedSet for hosts that were previously used for unpinned instances.
   • If there is no value set for NovaVcpuPinSet, then all Compute node cores should be assigned to either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, depending on the type of instances you intend to host on the nodes.

   For example, your previous environment file might contain the following pinning configuration:

   parameter_defaults:
     ...
     NovaVcpuPinSet: 1,2,3,5,6,7
     ...

   To migrate the configuration to a pinned configuration, set the NovaComputeCpuDedicatedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuDedicatedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   To migrate the configuration to an unpinned configuration, set the NovaComputeCpuSharedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuSharedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   Important

   Ensure the configuration of either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet matches the configuration defined in NovaVcpuPinSet. To change the configuration for either of these, or to configure both NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, ensure the Compute nodes with the pinning configuration are not running any instances before updating the configuration.

6. Save the file.

7. Run the deployment command to update the overcloud with the new CPU pinning parameters.

   (undercloud) $ openstack overcloud upgrade converge \
       --stack _STACK NAME_ \
       --templates \
       ...
       -e /home/stack/templates/<compute_environment_file>.yaml
       ...

Procedure

1. Correct the mistake in the file:

   parameter_defaults:
     UpgradeLeappEnabled: true
     UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms"
     CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Run the upgrade preparation command with the corrected file:

   $ openstack overcloud upgrade prepare \
       --stack STACK NAME \
       --templates \
       -e ENVIRONMENT FILE
       …​
       -e /home/stack/templates/upgrades-environment.yaml \
       …​

   Wait until the overcloud stack update completes.

3. Continue with the upgrade operation step that failed.