Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux (RHEL) repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos \
   --enable=rhel-8-for-x86_64-baseos-tus-rpms \
   --enable=rhel-8-for-x86_64-appstream-tus-rpms \
   --enable=rhel-8-for-x86_64-highavailability-tus-rpms \
   --enable=openstack-17.1-for-rhel-8-x86_64-rpms \
   --enable=fast-datapath-for-rhel-8-x86_64-rpms

3. Switch the container-tools module version to RHEL 8 on all nodes:

   [stack@director ~]$ sudo dnf -y module switch-to container-tools:rhel8

4. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Install the packages for validation:

   $ sudo dnf -y update openstack-tripleo-validations python3-validations-libs validations-common

4. Copy the inventory from mistral:

   $ sudo chown stack:stack /var/lib/mistral/.ssh/tripleo-admin-rsa
   $ sudo cat /var/lib/mistral/<stack>/tripleo-ansible-inventory.yaml > inventory.yaml

   • Replace <stack> with the name of the stack.

5. Run the validation:

   $ validation run -i inventory.yaml --group pre-upgrade

6. Review the script output to determine which validations succeed and fail:

   === Running validation: "check-ftype" ===
   
   Success! The validation passed for all hosts:
   * undercloud

Procedure

1. Log in to the undercloud host as the stack user.

2. Optional: Back up the 16.2 containers-prepare-parameter.yaml file:

   $ cp containers-prepare-parameter.yaml \
     containers-prepare-parameter.yaml.orig

3. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means that director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. Director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

4. Modify the containers-prepare-parameter.yaml to suit your requirements. For more information about container image parameters, see Container image preparation parameters.

5. If your deployment includes Red Hat Ceph Storage, update the Red Hat Ceph Storage container image parameters in the containers-prepare-parameter.yaml file for the version of Red Hat Ceph Storage that your deployment uses:

   ceph_namespace: registry.redhat.io/rhceph
   ceph_image: <ceph_image_file>
   ceph_tag: latest
   ceph_grafana_image: <grafana_image_file>
   ceph_grafana_namespace: registry.redhat.io/rhceph
   ceph_grafana_tag: latest

   • Replace <ceph_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • Red Hat Ceph Storage 5: rhceph-5-rhel8

   • Replace <grafana_image_file> with the name of the image file for the version of Red Hat Ceph Storage that your deployment uses:

     • Red Hat Ceph Storage 5: rhceph-5-dashboard-rhel8

Procedure

1. Log in to your undercloud host as the stack user.

2. Create a file called skip_rhel_release.yaml and set the SkipRhelEnforcement parameter to true:

   parameter_defaults:
     SkipRhelEnforcement: true

3. Open the undercloud.conf file and add the following parameters to the DEFAULT section in the file:

   container_images_file = /home/stack/containers-prepare-parameter.yaml
   custom_env_files = /home/stack/skip_rhel_release.yaml

   • Add any additional custom environment files to the custom_env_files parameter.

     The custom_env_files parameter defines the location of the skip_rhel_release.yaml file that is required for the upgrade.

   • The container_images_file parameter defines the location of the containers-prepare-parameter.yaml environment file so that director pulls container images for the undercloud from the correct location.
4. Check all other parameters in the file for any changes.
5. Save the file.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to disable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the metalsmith list command.
4. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the upgrade process.

Procedure

1. Select an environment file and the check if it has an ExtraConfig parameter:

   $ grep ExtraConfig ~/templates/custom-config.yaml

2. If the results show an ExtraConfig parameter for any role (e.g. ControllerExtraConfig) in the chosen file, check the full parameter structure in that file.

3. If the parameter contains any puppet Hierdata with a SECTION/parameter syntax followed by a value, it might have been been replaced with a parameter with an actual Puppet class. For example:

   parameter_defaults:
     ExtraConfig:
       neutron::config::dhcp_agent_config:
         'DEFAULT/dnsmasq_local_resolv':
           value: 'true'

4. Check the director’s Puppet modules to see if the parameter now exists within a Puppet class. For example:

   $ grep dnsmasq_local_resolv

   If so, change to the new interface.

5. The following are examples to demonstrate the change in syntax:

   • Example 1:

     parameter_defaults:
       ExtraConfig:
         neutron::config::dhcp_agent_config:
           'DEFAULT/dnsmasq_local_resolv':
             value: 'true'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         neutron::agents::dhcp::dnsmasq_local_resolv: true

   • Example 2:

     parameter_defaults:
       ExtraConfig:
         ceilometer::config::ceilometer_config:
           'oslo_messaging_rabbit/rabbit_qos_prefetch_count':
             value: '32'

     Changes to:

     parameter_defaults:
       ExtraConfig:
         oslo::messaging::rabbit::rabbit_qos_prefetch_count: '32'

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Verify that the following files that were exported during the undercloud upgrade contain the expected configuration for the overcloud upgrade. You can find the following files in the ~/overcloud-deploy directory:

   • tripleo-<stack>-passwords.yaml
   • tripleo-<stack>-network-data.yaml
   • tripleo-<stack>-virtual-ips.yaml

   • tripleo-<stack>-baremetal-deployment.yaml

     Note

     If these files were not generated after the undercloud upgrade, contact Red Hat Support.

4. On the main stack, copy the passwords.yaml file to the ~/overcloud-deploy/$(<stack>) directory. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-passwords.yaml ~/overcloud-deploy/<stack>/<stack>-passwords.yaml

   • Replace <stack> with the name of your stack.

5. On the main stack, copy the network-data.yaml file to the stack user’s home directory and deploy the networks. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-network-data.yaml ~/
   $ mkdir ~/overcloud_adopt
   $ openstack overcloud network provision --debug \
   --output /home/stack/overcloud_adopt/generated-networks-deployed.yaml tripleo-<stack>-network-data.yaml

   For more information, see Provisioning and deploying your overcloud in Installing and managing Red Hat OpenStack Platform with director.

6. On the main stack, copy the virtual-ips.yaml file to the stack user’s home directory and provision the network VIPs. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-virtual-ips.yaml ~/
   $ openstack overcloud network vip provision --debug \
   --stack <stack> --output \
   /home/stack/overcloud_adopt/generated-vip-deployed.yaml tripleo-<stack>-virtual-ips.yaml

7. On the main stack, copy the baremetal-deployment.yaml file to the stack user’s home directory and provision the overcloud nodes. Repeat this step on each stack in your environment:

   $ cp ~/overcloud-deploy/<stack>/tripleo-<stack>-baremetal-deployment.yaml ~/
   $ openstack overcloud node provision --debug --stack <stack> \
   --output /home/stack/overcloud_adopt/baremetal-deployment.yaml \
   tripleo-<stack>-baremetal-deployment.yaml

8. Complete the following steps to prepare the containers:

   1. Back up the containers-prepare-parameter.yaml file that you used for the undercloud upgrade:

      $ cp containers-prepare-parameter.yaml \
      containers-prepare-parameter.yaml.orig

   2. Define the following environment variables before you run the script to update the containers-prepare-parameter.yaml file:

      • NAMESPACE: The namespace for the UBI9 images. For example, NAMESPACE='"namespace":"example.redhat.com:5002",'
      • EL8_NAMESPACE: The namespace for the UBI8 images.
      • NEUTRON_DRIVER: The driver to use and determine which OpenStack Networking (neutron) container to use. Set to the type of containers you used to deploy the original stack. For example, set to NEUTRON_DRIVER='"neutron_driver":"ovn",' to use OVN-based containers.
      • EL8_TAGS: The tags of the UBI8 images, for example, EL8_TAGS='"tag":"rhel_8_rhosp17.1",'.
      • EL9_TAGS: The tags of the UBI9 images, for example, EL9_TAGS='"tag":"rhel_9_rhosp17.1",'.

      • CONTROL_PLANE_ROLES: The list of control plane roles using the --role option, for example, --role ControllerOpenstack, --role Database, --role Messaging, --role Networker, --role CephStorage. To view the list of control plane roles in your environment, run the following command:

        $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
        /var/lib/mistral/$STACK/tripleo-ansible-inventory.yaml \
        | grep -vi compute

      • COMPUTE_ROLES: The list of Compute roles using the --role option, for example, --Compute-1. To view the list of Compute roles in your environment, run the following command:

        $ sudo awk '/tripleo_role_name/ {print "--role " $2}' \
        /var/lib/mistral/$STACK/tripleo-ansible-inventory.yaml \
        | grep -i compute

      • CEPH_TAGS: If you deployed Red Hat Ceph Storage, specify Red Hat Ceph Storage 5 container images. For example, "ceph_tag":"5-404".

   3. Run the following script to to update the containers-prepare-parameter.yaml file:

      $ python3 /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \
           ${COMPUTE_ROLES} \
           ${CONTROL_PLANE_ROLES} \
           --enable-multi-rhel \
           --excludes collectd \
           --excludes nova-libvirt \
           --minor-override "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
           --major-override "{${EL9_TAGS}${NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
           --output-env-file \
          /home/stack/containers-prepare-parameter.yaml

9. If you are using Red Hat Ceph Storage, create a file called ceph_params.yaml and include the following content:

   parameter_defaults:
     CephSpecFqdn: true
     CephConfigPath: "/etc/ceph"
     CephAnsibleRepo: "rhceph-5-tools-for-rhel-8-x86_64-rpms"
     DeployedCeph: true

   Note

   If your Red Hat Ceph Storage deployment includes short names, you must set the CephSpecFqdn parameter to false. If set to true, the inventory generates with both the short names and domain names, causing the Red Hat Ceph Storage upgrade to fail.

10. Create an environment file called upgrades-environment.yaml in your templates directory and include the following content:

    parameter_defaults:
      ExtraConfig:
        nova::workarounds::disable_compute_service_check_for_ffu: true
      DnsServers: ["<dns_servers>"]
        DockerInsecureRegistryAddress: <undercloud_FQDN>
      UpgradeInitCommand: |
        sudo subscription-manager repos --disable=*
          if $( grep -q  9.2  /etc/os-release )
          then
            sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms
            sudo subscription-manager release --set=9.2
          else
            sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-tus-rpms --enable=rhel-8-for-x86_64-appstream-tus-rpms --enable=rhel-8-for-x86_64-highavailability-tus-rpms --enable=openstack-17.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
            sudo podman ps | grep -q ceph && subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
            sudo subscription-manager release --set=8.4
          fi
    
          if $(sudo podman ps | grep -q ceph )
          then
            sudo dnf -y install cephadm
          fi

    • Replace <dns_servers> with a comma-separated list of your DNS server IP addresses, for example, ["10.0.0.36", "10.0.0.37"].

    • Replace <undercloud_FQDN> with the fully qualified domain name (FQDN) of the undercloud host, for example, "undercloud-0.ctlplane.redhat.local:8787".

      For more information about the upgrade parameters that you can configure in the environment file, see Upgrade parameters.

11. On the undercloud, create a file called overcloud_upgrade_prepare.sh in your templates directory. You must create this file for each stack in your environment. This file includes the original content of your overcloud deploy file and the environment files that are relevant to your environment. For example:

    #!/bin/bash
    openstack overcloud upgrade prepare --yes \
      --timeout 460 \
      --templates /usr/share/openstack-tripleo-heat-templates \
      --ntp-server 192.168.24.1 \
      --stack <stack> \
      -r /home/stack/roles_data.yaml \
      -e /home/stack/templates/internal.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovs.yaml \
      -e /home/stack/templates/network/network-environment.yaml \
      -e /home/stack/templates/inject-trust-anchor.yaml \
      -e /home/stack/templates/hostnames.yml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
      -e /home/stack/templates/nodes_data.yaml \
      -e /home/stack/templates/debug.yaml \
      -e /home/stack/templates/firstboot.yaml \
      -e /home/stack/templates/upgrades-environment.yaml \
      -e /home/stack/overcloud-params.yaml \
      -e /home/stack/overcloud-deploy/overcloud/overcloud-network-environment.yaml \
      -e /home/stack/overcloud_adopt/baremetal-deployment.yaml \
      -e /home/stack/overcloud_adopt/generated-networks-deployed.yaml \
      -e /home/stack/overcloud_adopt/generated-vip-deployed.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/nova-hw-machine-type-upgrade.yaml \
      -e /home/stack/skip_rhel_release.yaml \
      -e ~/containers-prepare-parameter.yaml

    1. In the original network-environment.yaml file (/home/stack/templates/network/network-environment.yaml), remove all the resource_registry resources that point to OS::TripleO::*::Net::SoftwareConfig.

    2. In the overcloud_upgrade_prepare.sh file, include the following options relevant to your environment:

       • The environment file (upgrades-environment.yaml) with the upgrade-specific parameters (-e).
       • The environment file (containers-prepare-parameter.yaml) with your new container image locations (-e). In most cases, this is the same environment file that the undercloud uses.
       • The environment file (skip_rhel_release.yaml) with the release parameters (-e).
       • Any custom configuration environment files (-e) relevant to your deployment.
       • If applicable, your custom roles (roles_data) file using --roles-file.
       • For Ceph deployments, the environment file (ceph_params.yaml) with the Ceph parameters (-e).
       • The files that were generated during overcloud adoption (networks-deployed.yaml, vip-deployed.yaml,baremetal-deployment.yaml) (-e).
       • If applicable, the environment file (ipa-environment.yaml) with your IPA service (-e).

       • If you use a custom stack name, pass the name with the --stack option.

         Note

         You must include the nova-hw-machine-type-upgrade.yaml file in your templates until all of your RHEL 8 Compute nodes are upgraded to RHEL 9 in the environment. If this file is excluded, an error appears in the nova_compute.log in the /var/log/containers/nova directory. After you upgrade all of your RHEL 8 Compute nodes to RHEL 9, you can remove this file from your configuration and update the stack.

    3. If you enabled the Shared File Systems service (manila) with CephFS through NFS on the deployment that you are upgrading, you must specify an additional environment file at the end of the overcloud_upgrade_prepare.sh script file. You must add the environment file at the end of the script because it overrides another environment file that is specified earlier in the script:

       -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

12. Run the upgrade preparation command for each stack in your environment:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    $ sh /home/stack/overcloud_upgrade_prepare.sh

13. Wait until the upgrade preparation completes.

14. Download the container images:

    $ openstack overcloud external-upgrade run --stack <stack> --tags container_image_prepare

Procedure

1. Enable the Ceph 5 Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf update -y ceph-ansible

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the Red Hat Ceph Storage external upgrade process with the ceph tag:

   $ openstack overcloud external-upgrade run \
      --skip-tags "ceph_health,opendev-validation, ceph_ansible_remote_tmp" \
      --stack <stack> \
      --tags ceph,facts 2>&1

   • Replace <stack> with the name of your stack.

4. Create the ceph-admin user and distribute the appropriate keyrings:

   ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log \
   ANSIBLE_HOST_KEY_CHECKING=false \
   ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml \
     -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml \
    -e tripleo_admin_user=ceph-admin \
    -e distribute_private_key=true \
     --limit ceph_osd,ceph_mon,Undercloud

5. Update the packages on the Red Hat Ceph Storage nodes:

   $ openstack overcloud upgrade run \
       --stack <stack> \
       --skip-tags ceph_health,opendev-validation,ceph_ansible_remote_tmp \
       --tags setup_packages --limit ceph_osd,ceph_mon,Undercloud \
       --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1

   Note

   The Ceph Monitor service (CephMon) runs on Controller nodes. This command includes the ceph_mon tag, which also updates the packages on the Controller nodes.

6. Configure the Red Hat Ceph Storage nodes to use cephadm:

   $ openstack overcloud external-upgrade run \
       --skip-tags ceph_health,opendev-validation,ceph_ansible_remote_tmp \
       --stack <stack> \
       --tags cephadm_adopt  2>&1

7. Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file.

   #!/bin/bash
   openstack overcloud upgrade prepare  --yes \
     --timeout 460 \
    --templates /usr/share/openstack-tripleo-heat-templates \
     --ntp-server 192.168.24.1 \
     --stack <stack> \
     -r /home/stack/roles_data.yaml \
     -e /home/stack/templates/internal.yaml \
     …
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml \
     -e ~/containers-prepare-parameter.yaml

   Note

   This example uses the environments/cephadm/cephadm-rbd-only.yaml file because RGW is not deployed. If you plan to deploy RGW, use environments/cephadm/cephadm.yaml after you finish upgrading your RHOSP environment, and then run a stack update.

8. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation:

   -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml

9. Save the file.

10. Run the upgrade preparation command:

    $ source stackrc
    $ chmod 755 /home/stack/overcloud_upgrade_prepare.sh
    sh /home/stack/overcloud_upgrade_prepare.sh

    The Red Hat Ceph Storage cluster is now upgraded to version 5. This has the following implications:

    • You no longer use ceph-ansible to manage Red Hat Ceph Storage. Instead, the Ceph Orchestrator manages the Red Hat Ceph Storage cluster. For more information about the Ceph Orchestrator, see The Ceph Operations Guide.
    • You no longer need to perform stack updates to make changes to the Red Hat Ceph Storage cluster in most cases. Instead, you can run day two Red Hat Ceph Storage operations directly on the cluster as described in The Ceph Operations Guide. You can also scale Red Hat Ceph Storage cluster nodes up or down as described in Scaling the Ceph Storage cluster in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director.
    • Inspect the Red Hat Ceph Storage cluster’s health. For more information about monitoring your cluster’s health, see Monitoring Red Hat Ceph Storage nodes in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director.

    • Do not include environment files, such as environments/ceph-ansible/ceph-ansible.yaml, in openstack deployment commands such as openstack overcloud deploy. If your deployment includes ceph-ansible environment files, replace them with one of the following options:

      Red Hat Ceph Storage deployment	Original ceph-ansible file	Cephadm file replacement
      Ceph RADOS Block Device (RBD) only	Any ceph-ansible environment file	environments/cephadm/cephadm-rbd-only.yaml
      RBD and the Ceph Object Gateway (RGW)	Any ceph-ansible environment file	environments/cephadm/cephadm.yaml
      Ceph Dashboard	environments/ceph-ansible/ceph-dashboard.yaml	Respective file in environments/cephadm/
      Ceph MDS	environments/ceph-ansible/ceph-mds.yaml	Respective file in environments/cephadm/

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Upgrade RHOSP on all nodes in your main stack:

   $ openstack overcloud upgrade run --yes --stack <stack> --debug --limit allovercloud,undercloud --playbook all

   • Replace <stack> with the name of the overcloud stack that you want to upgrade the nodes on.

     Repeat this step for each stack in your RHOSP deployment.

Procedure

1. Log in to the undercloud as the stack user.

2. Check the /etc/ssh/sshd_config file for the PermitRootLogin parameter:

   $ sudo grep PermitRootLogin /etc/ssh/sshd_config

3. If the parameter is not in the /etc/ssh/sshd_config file, edit the file and set the PermitRootLogin parameter:

   PermitRootLogin no

4. Save the file.

Procedure

1. Validate your SSH key size:

   ssh-keygen -l -f ~/.ssh/id_rsa.pub

   Example output:

   1024 SHA256:Xqz0Xz0/aJua6B3qRD7VsLr6n/V3zhmnGSkcFR6FlJw stack@director.example.local (RSA)

2. If your SSH key is less than 2048 bits, use the ssh_key_rotation.yaml ansible playbook to replace the SSH key before continuing.

   ansible-playbook \
   -i ~/overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml \
   /usr/share/ansible/tripleo-playbooks/ssh_key_rotation.yaml \
   --extra-vars "keep_old_key_authorized_keys=<true|false> backup_folder_path=</path/to/backup_folder>"

   • Replace <stack> with the name of your overcloud stack.
   • Replace <true|false> based on your backup needs. If you do not include this variable, the default value is true.
   • Replace </path/to/backup_folder> with a backup path. If you do not include this variable, the default value is /home/{{ ansible_user_id }}/backup_keys/{{ ansible_date_time.epoch }}

Procedure

1. Log in to the undercloud as the stack user.

2. Create a file named system_upgrade.yaml in your templates directory and include the following content:

   parameter_defaults:
     UpgradeLeappDevelSkip: "LEAPP_UNSUPPORTED=1 LEAPP_DEVEL_SKIP_CHECK_OS_RELEASE=1 LEAPP_NO_NETWORK_RENAMING=1 LEAPP_DEVEL_TARGET_RELEASE=9.2"
     UpgradeLeappDebug: false
     UpgradeLeappEnabled: true
     LeappActorsToRemove: ['checkifcfg','persistentnetnamesdisable','checkinstalledkernels','biosdevname']
     LeappRepoInitCommand: |
        sudo subscription-manager repos --disable=*
        subscription-manager repos --enable rhel-8-for-x86_64-baseos-tus-rpms --enable rhel-8-for-x86_64-appstream-tus-rpms --enable openstack-17.1-for-rhel-8-x86_64-rpms
         subscription-manager release --set=8.4
     UpgradeLeappCommandOptions:
   "--enablerepo=rhel-9-for-x86_64-baseos-eus-rpms --enablerepo=rhel-9-for-x86_64-appstream-eus-rpms --enablerepo=rhel-9-for-x86_64-highavailability-eus-rpms --enablerepo=openstack-17.1-for-rhel-9-x86_64-rpms --enablerepo=fast-datapath-for-rhel-9-x86_64-rpms"

   Note

   If your deployment includes Red Hat Ceph Storage nodes, you must add the CephLeappRepoInitCommand parameter and specify the source OS version of your Red Hat Ceph Storage nodes. For example:

   CephLeappRepoInitCommand:
   ...
     subscription-manager release --set=8.6

3. Add the LeappInitCommand parameter to your system_upgrade.yaml file to specify additional requirements applicable to your environment, for example, if you need to define role-based overrides:

     LeappInitCommand: |
       sudo subscription-manager repos --disable=*
       sudo subscription-manager repos
     --enable=rhel-9-for-x86_64-baseos-eus-rpms --enable=rhel-9-for-x86_64-appstream-eus-rpms --enable=rhel-9-for-x86_64-highavailability-eus-rpms --enable=openstack-17.1-for-rhel-9-x86_64-rpms --enable=fast-datapath-for-rhel-9-x86_64-rpms
   
       leapp answer --add --section check_vdo.confirm=True
   
       dnf -y remove irb

4. If you use kernel-based NIC names, add the following parameter to the system_upgrade.yaml file to ensure that the NIC names persist throughout the upgrade process:

   parameter_defaults:
     NICsPrefixesToUdev: ['en']
   ...

5. Run the Leapp upgrade:

   $ openstack undercloud upgrade --yes --system-upgrade \
   /home/stack/system_upgrade.yaml

   Note

   If you need to run the Leapp upgrade again, you must first reset the repositories to RHEL 8.

6. Reboot the undercloud:

   $ sudo reboot

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the following script without the CONTROL_PLANE_ROLES parameter. Ensure that you include the variables that you used to prepare the containers in Running the overcloud upgrade preparation.

   python3 \
   /usr/share/openstack-tripleo-heat-templates/tools/multi-rhel-container-image-prepare.py \
        ${COMPUTE_ROLES} \
        --enable-multi-rhel \
        --excludes collectd \
        --excludes nova-libvirt \
        --minor-override \
   "{${EL8_TAGS}${EL8_NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
        --major-override \
        "{${EL9_TAGS}${NAMESPACE}${CEPH_TAGS}${NEUTRON_DRIVER}\"no_tag\":\"not_used\"}" \
        --output-env-file \
   /home/stack/containers-prepare-parameter.yaml

   Note

   The CONTROL_PLANE_ROLES parameter defines the list of your control plane roles. Removing this parameter from the script prepares the control plane roles for an upgrade to RHEL 9.2. If the CONTROL_PLANE_ROLES parameter is included in the script, the control plane roles remain on RHEL 8.4.

4. In the skip_rhel_release.yaml file, set the SkipRhelEnforcement parameter to false:

   parameter_defaults:
     SkipRhelEnforcement: false

5. Update the overcloud_upgrade_prepare.sh file:

   $ openstack overcloud upgrade prepare --yes \
       ...
       -e /home/stack/system_upgrade.yaml \
       -e /home/stack/containers-prepare-parameter.yaml \
       -e /home/stack/skip_rhel_release.yaml \
       ...

   • Include the system_upgrade.yaml file with the upgrade-specific parameters (-e).
   • Include the containers-prepare-parameter.yaml file with the control plane roles removed (-e).
   • Include the skip_rhel_release.yaml file with the release parameters (-e).

6. Run the overcloud_upgrade_prepare.sh script:

   $ sh /home/stack/overcloud_upgrade_prepare.sh

7. Fetch any new or modified containers that you require for the system upgrade:

   $ openstack overcloud external-upgrade run  \
        --stack <stack> \
        --tags container_image_prepare 2>&1

8. Upgrade the first one-third of the control plane nodes:

   $ openstack overcloud upgrade run --yes \
        --stack <stack> \
        --tags system_upgrade \
        --limit <controller-0>,<database-0>,<messaging-0>,<networker-0>,<ceph-0>

   • Replace <stack> with the name of your stack.
   • Replace <controller-0>,<database-0>,<messaging-0>,<networker-0>,<ceph-0> with your own node names.

9. Log in to each upgraded node and verify that the cluster in each node is running:

   $ sudo pcs status

   Repeat this verification step after you upgrade the second one-third of your control plane nodes, and after you upgrade the last one-third of your control plane nodes.

10. Upgrade the second one-third of the control plane nodes:

    $ openstack overcloud upgrade run --yes \
         --stack <stack> \
         --tags system_upgrade \
         --limit <controller-1>,<database-1>,<messaging-1>,<networker-1>,<ceph-1>

    • Replace <controller-1>,<database-1>,<messaging-1>,<networker-1>,<ceph-1> with your own node names.

11. Upgrade the last one-third of the control plane nodes:

    $ openstack overcloud upgrade run --yes \
         --stack <stack> \
         --tags system_upgrade \
         --limit <controller-2>,<database-2>,<messaging-2>,<networker-2>,<ceph-2>

    • Replace <controller-2>,<database-2>,<messaging-2>,<networker-2>,<ceph-2> with your own node names.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In the container-image-prepare.yaml file, ensure that only the tags specified in the ContainerImagePrepare parameter are included, and the MultiRhelRoleContainerImagePrepare parameter is removed. For example:

   parameter_defaults:
     ContainerImagePrepare:
     - tag_from_label: "{version}-{release}"
       set:
         namespace:
         name_prefix:
         name_suffix:
         tag:
         rhel_containers: false
         neutron_driver: ovn
         ceph_namespace:
         ceph_image:
         ceph_tag:

4. In the roles_data.yaml file, replace the OS::TripleO::Services::NovaLibvirtLegacy service with the OS::TripleO::Services::NovaLibvirt service that is required for RHEL 9.2.

5. Run the openstack overcloud upgrade prepare command and include the system_upgrade.yaml file with the upgrade-specific parameters:

   $ openstack overcloud upgrade prepare --yes
   …​
   -e /home/stack/system_upgrade.yaml
   …​

6. Upgrade the operating system on the Compute nodes to RHEL 9.2. Use the --limit option with a comma-separated list of nodes that you want to upgrade. The following example upgrades the compute-0, compute-1, and compute-2 nodes.

   $ openstack overcloud upgrade run --yes --tags system_upgrade --stack <stack> --limit compute-0,compute-1,compute-2

   • Replace <stack> with the name of your stack.

7. Upgrade the containers on the Compute nodes to RHEL 9.2. Use the --limit option with a comma-separated list of nodes that you want to upgrade. The following example upgrades the compute-0, compute-1, and compute-2 nodes.

   $ openstack overcloud upgrade run --yes --stack <stack>  --limit compute-0,compute-1,compute-2

1. Plan which nodes you want to upgrade to RHEL 9.2, and which nodes you want to remain on RHEL 8.4. Choose a role name for each role that you are creating for each batch of nodes, for example, ComputeRHEL-9.2 and ComputeRHEL-8.4.

2. Create roles that store the nodes that you want to upgrade to RHEL 9.2, or the nodes that you want to stay on RHEL 8.4. These roles can remain empty until you are ready to move your Compute nodes to a new role. You can create as many roles as you need and divide nodes among them any way you decide. For example:

   • If your environment uses a role called ComputeSRIOV and you need to run a canary test to upgrade to RHEL 9.2, you can create a new ComputeSRIOVRHEL9 role and move the canary node to the new role.
   • If your environment uses a role called ComputeOffload and you want to upgrade most nodes in that role to RHEL 9.2, but keep a few nodes on RHEL 8.4, you can create a new ComputeOffloadRHEL8 role to store the RHEL 8.4 nodes. You can then select the nodes in the original ComputeOffload role to upgrade to RHEL 9.2.
3. Move the nodes from each Compute role to the new role.

4. Upgrade the operating system on specific Compute nodes to RHEL 9.2. You can upgrade nodes in batches from the same role or multiple roles.

   Note

   In a Multi-RHEL environment, the deployment should continue to use the pc-i440fx machine type. Do not update the default to Q35. Migrating to the Q35 machine type is a separate, post-upgrade procedure to follow after all Compute nodes are upgraded to RHEL 9.2. For more information about migrating the Q35 machine type, see Updating the default machine type for hosts after an upgrade to RHOSP 17.

   Use the following procedures to upgrade Compute nodes to a Multi-RHEL environment:

   • Creating roles for Multi-RHEL Compute nodes
   • Upgrading the Compute node operating system

Procedure

1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

2. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-17.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-17.1.tar; do tar -xvf $i; done
   $ cd ~

Procedure

1. Log in to the undercloud as the stack user.

2. If your Compute nodes support simultaneous multithreading (SMT) but you created instances with the hw:cpu_thread_policy=isolated policy, you must perform one of the following options:

   • Unset the hw:cpu_thread_policy thread policy and resize the instances:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Unset the hw:cpu_thread_policy property of the flavor:

        (overcloud) $ openstack flavor unset --property hw:cpu_thread_policy <flavor>

        Note

        • Unsetting the hw:cpu_thread_policy attribute sets the policy to the default prefer policy, which sets the instance to use an SMT-enabled Compute node if available. You can also set the hw:cpu_thread_policy attribute to require, which sets a hard requirements for an SMT-enabled Compute node.
        • If the Compute node does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling will fail. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The default prefer policy ensures that thread siblings are used when available.
        • If you use hw:cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

     3. Convert the instances to use the new thread policy.

        (overcloud) $ openstack server resize --flavor <flavor> <server>
        (overcloud) $ openstack server resize confirm <server>

        Repeat this step for all pinned instances using the hw:cpu_thread_policy=isolated policy.

   • Migrate instances from the Compute node and disable SMT on the Compute node:

     1. Source your overcloud authentication file:

        $ source ~/overcloudrc

     2. Disable the Compute node from accepting new virtual machines:

        (overcloud) $ openstack compute service list
        (overcloud) $ openstack compute service set <hostname> nova-compute --disable

     3. Migrate all instances from the Compute node. For more information on instance migration, see Migrating virtual machine instances between Compute nodes.
     4. Reboot the Compute node and disable SMT in the BIOS of the Compute node.
     5. Boot the Compute node.

     6. Re-enable the Compute node:

        (overcloud) $ openstack compute service set <hostname> nova-compute --enable

3. Source the stackrc file:

   $ source ~/stackrc

4. Edit the environment file that contains the NovaVcpuPinSet parameter.

5. Migrate the CPU pinning configuration from the NovaVcpuPinSet parameter to NovaComputeCpuDedicatedSet and NovaComputeCpuSharedSet:

   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuDedicatedSet for hosts that were previously used for pinned instances.
   • Migrate the value of NovaVcpuPinSet to NovaComputeCpuSharedSet for hosts that were previously used for unpinned instances.
   • If there is no value set for NovaVcpuPinSet, then all Compute node cores should be assigned to either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, depending on the type of instances you intend to host on the nodes.

   For example, your previous environment file might contain the following pinning configuration:

   parameter_defaults:
     ...
     NovaVcpuPinSet: 1,2,3,5,6,7
     ...

   To migrate the configuration to a pinned configuration, set the NovaComputeCpuDedicatedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuDedicatedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   To migrate the configuration to an unpinned configuration, set the NovaComputeCpuSharedSet parameter and unset the NovaVcpuPinSet parameter:

   parameter_defaults:
     ...
     NovaComputeCpuSharedSet: 1,2,3,5,6,7
     NovaVcpuPinSet: ""
     ...

   Important

   Ensure the configuration of either NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet matches the configuration defined in NovaVcpuPinSet. To change the configuration for either of these, or to configure both NovaComputeCpuDedicatedSet or NovaComputeCpuSharedSet, ensure the Compute nodes with the pinning configuration are not running any instances before updating the configuration.

6. Save the file.

7. Run the deployment command to update the overcloud with the new CPU pinning parameters.

   (undercloud) $ openstack overcloud deploy \
       --stack _STACK NAME_ \
       --templates \
       ...
       -e /home/stack/templates/<compute_environment_file>.yaml
       ...

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file.

   $ source ~/stackrc

3. Log in to a Controller node as the heat-admin user:

   (undercloud)$ metalsmith list
   $ ssh heat-admin@<controller_ip>

   Replace <controller_ip> with the IP address of the Controller node.

4. Retrieve the list of instances that have no machine type set:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt list_unset_machine_type

5. Check the NovaHWMachineType parameter in the nova-hw-machine-type-upgrade.yaml file for the default machine type for the instance host. The default value for the NovaHWMachineType parameter in RHOSP 16.2 is as follows:

   x86_64=pc-i440fx-rhel7.6.0,aarch64=virt-rhel7.6.0,ppc64=pseries-rhel7.6.0,ppc64le=pseries-rhel7.6.0

6. Update the system metadata of each instance with the default instance machine type:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt update_machine_type <instance_uuid> <machine_type>

   • Replace <instance_uuid> with the UUID of the instance.
   • Replace <machine_type> with the machine type to record for the instance.

7. Confirm that the machine type is recorded for all instances:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-status upgrade check

   This command returns a warning if an instance is found without a machine type. If you get this warning, repeat this procedure from step 4.

8. Change the default value of NovaHWMachineType in a Compute environment file to x86_64=q35 and deploy the overcloud.

Verification

1. Create an instance that has the default machine type:

   (overcloud)$ openstack server create --flavor <flavor> \
     --image <image> --network <network> \
     --wait defaultMachineTypeInstance

   • Replace <flavor> with the name or ID of a flavor for the instance.
   • Replace <image> with the name or ID of an image that does not set hw_machine_type.
   • Replace <network> with the name or ID of the network to connect the instance to.

2. Verify that the instance machine type is set to the default value:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt get_machine_type <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

3. Hard reboot an instance with a machine type of x86_64=pc-i440fx:

   (overcloud)$ openstack server reboot --hard <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

4. Verify that the instance machine type has not been changed:

   [heat-admin@<controller_ip> ~]$ sudo podman exec -i -u root nova_api \
     nova-manage libvirt get_machine_type <instance_uuid>

   Replace <instance_uuid> with the UUID of the instance.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Log in to a Controller node and run the Pacemaker command to re-enable fencing:

   $ ssh tripleo-admin@<controller_ip> "sudo pcs property set stonith-enabled=true"

   • Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command.
4. In the fencing.yaml environment file, set the EnableFencing parameter to true.