Keeping Red Hat OpenStack Platform Updated

Red Hat OpenStack Platform 16.2

Performing minor updates of Red Hat OpenStack Platform

OpenStack Documentation Team

Abstract

You can perform a minor update of your Red Hat OpenStack Platform (RHOSP) environment to keep it updated with the latest packages and containers.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Providing feedback on Red Hat documentation

We appreciate your input on our documentation. Tell us how we can make it better.

Using the Direct Documentation Feedback (DDF) function

Use the Add Feedback DDF function for direct comments on specific sentences, paragraphs, or code blocks.

  1. View the documentation in the Multi-page HTML format.
  2. Ensure that you see the Feedback button in the upper right corner of the document.
  3. Highlight the part of text that you want to comment on.
  4. Click Add Feedback.
  5. Complete the Add Feedback field with your comments.
  6. Optional: Add your email address so that the documentation team can contact you for clarification on your issue.
  7. Click Submit.

Chapter 1. Preparing for a minor update

Keep your Red Hat OpenStack Platform (RHOSP) 16.2 environment updated with the latest packages and containers.

The "The RHOSP minor update process workflow" outlines the steps you must complete to update your RHOSP environment.

The RHOSP minor update process workflow

  1. Prepare your environment for the RHOSP minor update.
  2. Update the undercloud to the latest OpenStack 16.2.z version.
  3. Update the overcloud to the latest OpenStack 16.2.z version.
  4. Upgrade all Ceph Storage services.
  5. Run the convergence command to refresh your overcloud stack.

If you have a multistack infrastructure, update each overcloud stack completely, one at a time. If you have a distributed compute node (DCN) infrastructure, update the overcloud at the central location completely, and then update the overcloud at each edge site, one at a time.

Prerequisites

  • Before you can update your environment from RHOSP 16.1 to RHOSP 16.2, ensure that your environment is updated to the latest minor release of RHOSP 16.1. For more information, see the 16.1 Keeping Red Hat OpenStack Platform Updated guide.

    You can update the following versions:

    Old RHOSP VersionNew RHOSP Version

    Red Hat OpenStack Platform 16.1.z latest

    Red Hat OpenStack Platform 16.2

  • Familiarize yourself with the known issues that might block an update.

Known issues that might block an update

Review the following known issues that might affect a successful minor version update.

BZ#1975240 - update from 16.1 to 16.2, when enabling tsx flag, compute node get restarted during update and ping loss occurs

Starting with Red Hat Enterprise Linux (RHEL) version 8.3, support for the Intel Transactional Synchronization Extensions (TSX) feature is disabled by default. This causes issues with instance live migration between hosts in the following migration scenario:

  • Migrating from hosts where the TSX kernel argument is enabled to hosts where the TSX kernel argument is disabled.

Live migration can be unsuccessful in Intel hosts that support the TSX feature. For more information about the CPUs that are affected by this issue, see Affected Configurations.

For more information, review the following Red Hat Knowledgebase solution Guidance on Intel TSX impact on OpenStack guests.

BZ#1973660 - (update) from 16.1 to 16.2 breaks trying to configure the rabbitmq service.

Overcloud nodes that run Pacemaker version 2.0.3-5.el8_2.4 might fail to update successfully because of a race condition that occurs when shutting down the cluster on a node.

If Pacemaker version 2.0.3-5.el8_2.4 is currently installed on any of the overcloud nodes, to avoid BZ#1973660, you must upgrade Pacemaker before you can update the overcloud nodes. For more information, see the following Red Hat Knowledgebase solution Update from OSP16.1 to OSP16.2 might fail to update certain HA containers.

BZ#1872404 - restarting nodes in parallel while maintaining quorum creates an unexpected node shutdown
Until this issue is resolved, for nodes based on composable roles, you must update the Database role first, before you can update Controller, Messaging, Compute, Ceph, and other roles.

Procedure

To prepare your RHSOP environment for the minor update, complete the following procedures:

1.1. Locking the environment to a Red Hat Enterprise Linux release

Red Hat OpenStack Platform (RHOSP) 16.2 is supported on Red Hat Enterprise Linux (RHEL) 8.4. Before you perform the update, lock the undercloud and overcloud repositories to the RHEL 8.4 release to avoid upgrading the operating system to a newer minor release.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check if your subscription management configuration includes the rhsm_release parameter. If the rhsm_release parameter is not present, add it and set it to 8.4:

    parameter_defaults:
      RhsmVars:
        …​
        rhsm_username: "myusername"
        rhsm_password: "p@55w0rd!"
        rhsm_org_id: "1234567"
        rhsm_pool_ids: "1a85f9223e3d5e43013e3d6e8ff506fd"
        rhsm_method: "portal"
        rhsm_release: "8.4"
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to lock the operating system version to RHEL 8.4 on all nodes:

    $ cat > ~/set_release.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: set release to 8.4
          command: subscription-manager release --set=8.4
          become: true
    EOF
  8. Run the set_release.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/set_release.yaml --limit undercloud,Controller,Compute

    Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Ceph Storage nodes because you are most likely using a different subscription for these nodes.

Note

To manually lock a node to a version, log in to the node and run the subscription-manager release command:

$ sudo subscription-manager release --set=8.4

1.2. Changing to Extended Update Support (EUS) repositories

Your Red Hat OpenStack Platform (RHOSP) subscription includes repositories for Red Hat Enterprise Linux (RHEL) 8.4 Extended Update Support (EUS). The EUS repositories include the latest security patches and bug fixes for RHEL 8.4. Switch to the following repositories before you perform an update.

Table 1.1. EUS repositories for RHEL 8.4

Standard repositoryEUS repository

rhel-8-for-x86_64-baseos-rpms

rhel-8-for-x86_64-baseos-eus-rpms

rhel-8-for-x86_64-appstream-rpms

rhel-8-for-x86_64-appstream-eus-rpms

rhel-8-for-x86_64-highavailability-rpms

rhel-8-for-x86_64-highavailability-eus-rpms

Important

You must use EUS repositories to retain compatibility with a specific version of Podman. Later versions of Podman are untested with Red Hat OpenStack Platform 16.2 and can cause unexpected results.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check the rhsm_repos parameter in your subscription management configuration. If this parameter does not include the EUS repositories, change the relevant repositories to the EUS versions:

    parameter_defaults:
      RhsmVars:
        rhsm_repos:
          *- rhel-8-for-x86_64-baseos-eus-rpms*
          *- rhel-8-for-x86_64-appstream-eus-rpms*
          *- rhel-8-for-x86_64-highavailability-eus-rpms*
          - ansible-2.9-for-rhel-8-x86_64-rpms
          - advanced-virt-for-rhel-8-x86_64-rpms
          - openstack-16.2-for-rhel-8-x86_64-rpms
          - rhceph-4-tools-for-rhel-8-x86_64-rpms
          - fast-datapath-for-rhel-8-x86_64-rpms
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to set the repositories to RHEL 8.4 EUS on all nodes:

    $ cat > ~/change_eus.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change to eus repos
          command: subscription-manager repos --disable=rhel-8-for-x86_64-baseos-rpms --disable=rhel-8-for-x86_64-appstream-rpms --disable=rhel-8-for-x86_64-highavailability-rpms --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms
          become: true
    EOF
  8. Run the change_eus.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/change_eus.yaml --limit undercloud,Controller,Compute

    Use the --limit option to apply the content to all Red Hat OpenStack Platform nodes. Do not run this playbook against Ceph Storage nodes because they usually use a different subscription.

1.3. Updating Red Hat Openstack Platform and Ansible repositories

Update your repositories to use Red Hat OpenStack Platform (RHOSP) 16.2 and Ansible 2.9 packages.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your overcloud subscription management environment file, which is the file that contains the RhsmVars parameter. The default name for this file is usually rhsm.yml.
  4. Check the rhsm_repos parameter in your subscription management configuration. If the rhsm_repos parameter is using the RHOSP 16.1 and Ansible 2.9 repositories, change the repository to the correct versions:

    parameter_defaults:
      RhsmVars:
        rhsm_repos:
          - rhel-8-for-x86_64-baseos-eus-rpms
          - rhel-8-for-x86_64-appstream-eus-rpms
          - rhel-8-for-x86_64-highavailability-eus-rpms
          *- ansible-2.9-for-rhel-8-x86_64-rpms*
          - advanced-virt-for-rhel-8-x86_64-rpms
          *- openstack-16.2-for-rhel-8-x86_64-rpms*
          - fast-datapath-for-rhel-8-x86_64-rpms
  5. Save the overcloud subscription management environment file.
  6. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  7. Create a playbook that contains a task to set the repositories to RHOSP16.2 on all nodes:

    $ cat > ~/update_rhosp_repos.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change osp repos
          command: subscription-manager repos --disable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=openstack-16.2-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms
          become: true
    EOF
  8. Run the update_rhosp_repos.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_rhosp_repos.yaml --limit undercloud,Controller,Compute

    Use the --limit option to apply the content to all RHOSP nodes. Do not run this playbook against Ceph Storage nodes because they usually use a different subscription.

  9. Create a playbook that contains a task to set the repositories to RHOSP 16.2 on all nodes:

    $ cat > ~/update_ceph_repos.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: change ceph repos
          command: subscription-manager repos --disable=openstack-16-deployment-tools-for-rhel-8-x86_64-rpms --enable=openstack-16.2-deployment-tools-for-rhel-8-x86_64-rpms --disable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms
          become: true
    EOF
  10. Run the update_ceph_repos.yaml playbook:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/update_ceph_repos.yaml --limit CephStorage

    Use the --limit option to apply the content to Ceph Storage nodes.

1.4. Setting the container-tools and virt module versions

Set the container-tools module to version 3.0 and the virt module to av to ensure that you use the correct package versions on all nodes.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Create a static inventory file of your overcloud:

    $ tripleo-ansible-inventory --ansible_ssh_user heat-admin --static-yaml-inventory ~/inventory.yaml

    If you use an overcloud name that is different to the default overcloud name of overcloud, set the name of your overcloud with the --plan option.

  4. Create a playbook that contains a task to set the container-tools module to version 3.0 on all nodes:

    $ cat > ~/container-tools.yaml <<'EOF'
    - hosts: all
      gather_facts: false
      tasks:
        - name: disable default dnf module for container-tools
          command: dnf module disable -y container-tools:rhel8
          become: true
        - name: set dnf module for container-tools:3.0
          command: dnf module enable -y container-tools:3.0
          become: true
    
    - hosts: undercloud,Compute,Controller
      gather_facts: false
      tasks:
        - name: disable default dnf module for virt
          command: dnf module disable -y virt:rhel
          become: true
        - name: disable 8.2 dnf module for virt
          command: dnf module disable -y virt:8.2
          become: true
        - name: set dnf module for virt:av
          command: dnf module enable -y virt:av
          become: true
    EOF
  5. Run the container-tools.yaml playbook against all nodes:

    $ ansible-playbook -i ~/inventory.yaml -f 25 ~/container-tools.yaml

1.5. Updating the container image preparation file

The container preparation file is the file that contains the ContainerImagePrepare parameter. You use this file to define the rules for obtaining container images for the undercloud and overcloud.

Before you update your environment, check the file to ensure that you obtain the correct image versions.

Procedure

  1. Edit the container preparation file. The default name for this file is usually containers-prepare-parameter.yaml.
  2. Check the tag parameter is set to 16.2 for each rule set:

    parameter_defaults:
      ContainerImagePrepare:
      - push_destination: true
        set:
          ...
          *tag: '16.2'*
        tag_from_label: '{version}-{release}'
    Note

    If you do not want to use a specific tag for the update, such as 16.2 or 16.2.2, remove the tag key-value pair and specify tag_from_label only. This uses the installed Red Hat OpenStack Platform version to determine the value for the tag to use as part of the update process.

  3. Save this file.

1.6. Updating the SSL/TLS configuration

Remove the NodeTLSData resource from the resource_registry to update your SSL/TLS configuration.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Edit your custom overcloud SSL/TLS public endpoint file, which is usually named ~/templates/enable-tls.yaml.
  4. Remove the NodeTLSData resource from the resource_registry:

    resource_registry:
      *OS::TripleO::NodeTLSData: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/tls/tls-cert-inject.yaml*
      ...

    The overcloud deployment uses a new service in HAProxy to determine if SSL/TLS is enabled.

    Note

    If this is the only resource in the resource_registry section of the enable-tls.yaml file, remove the complete resource_registry section.

  5. Save the SSL/TLS public endpoint file file.

1.7. Disabling fencing in the overcloud

Before you update the overcloud, ensure that fencing is disabled.

If fencing is deployed in your environment during the Controller nodes update process, the overcloud might detect certain nodes as disabled and attempt fencing operations, which can cause unintended results.

If you have enabled fencing in the overcloud, you must temporarily disable fencing for the duration of the update to avoid any unintended results.

Note

To re-enable fencing in your overcloud, set the EnableFencing parameter to true in the fencing.yaml environment file before you run the openstack overcloud update converge command. Director enables fencing in your overcloud during the deployment.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file.

    $ source ~/stackrc
  3. Log in to a Controller node and run the Pacemaker command to disable fencing:

    $ ssh heat-admin@<controller_ip> "sudo pcs property set stonith-enabled=false"

    Replace <controller_ip> with the IP address of a Controller node. You can find the IP addresses of your Controller nodes with the openstack server list command.

  4. In the fencing.yaml environment file, set the EnableFencing parameter to false to ensure that fencing stays disabled during the update process.

Chapter 2. Updating the undercloud

You can use director to update the main packages on the undercloud node. To update the undercloud and its overcloud images to the latest Red Hat OpenStack Platform (RHOSP) 16.2 version, complete the following procedures:

Prerequisites

  • Before you can update the undercloud to the latest RHSOP 16.2 version, ensure that you complete all the update preparation procedures. For more information, see Chapter 1, Preparing for a minor update

2.1. Performing a minor update of a containerized undercloud

Director provides commands to update the main packages on the undercloud node. Use director to perform a minor update within the current version of your RHOSP environment.

Procedure

  1. On the undercloud node, log in as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. Update the director main packages with the dnf update comand:

    $ sudo dnf update -y python3-tripleoclient* tripleo-ansible ansible
  4. Update the undercloud environment with the openstack undercloud upgrade command :

    $ openstack undercloud upgrade
  5. Wait until the undercloud update process completes.
  6. Reboot the undercloud to update the operating system’s kernel and other system packages:

    $ sudo reboot
  7. Wait until the node boots.

2.2. Updating the overcloud images

You must replace your current overcloud images with new versions to ensure that director can introspect and provision your nodes with the latest version of the RHOSP software.

Prerequisites

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

    $ rm -rf ~/images/*
  3. Extract the archives:

    $ cd ~/images
    $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-{ucversion}.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-{ucversion}.tar; do tar -xvf $i; done
    $ cd ~
  4. Import the latest images into the director:

    $ openstack overcloud image upload --update-existing --image-path /home/stack/images/
  5. Configure your nodes to use the new images:

    $ openstack overcloud node configure $(openstack baremetal node list -c <UUID> -f value)
  6. Verify the existence of the new images:

    $ openstack image list
    $ ls -l /var/lib/ironic/httpboot
Important

When you deploy overcloud nodes, ensure that the overcloud image version corresponds to the respective heat template version. For example, use only the RHOSP 16.2 images with the RHOSP 16.2 heat templates.

Important

The new overcloud-full image replaces the old overcloud-full image. If you made changes to the old image, you must repeat the changes in the new image, especially if you want to deploy new nodes in the future.

Chapter 3. Updating the overcloud

After you update the undercloud, you can update the overcloud by running the overcloud and container image preparation commands, updating your nodes, and running the overcloud update converge command. The control plane API is fully available during a minor update.

Prerequisites

  • You have updated the undercloud node to the latest version. For more information, see Chapter 2, Updating the undercloud.
  • If you use a local set of core templates in your stack user home directory, ensure that you update the templates and use the recommended workflow in Using Customized Core Heat Templates in the Advanced Overcloud Customization guide. You must update the local copy before you upgrade the overcloud.

Procedure

To update the overcloud, you must complete the following procedures:

3.1. Running the overcloud update preparation

To prepare the overcloud for the update process, you must run the openstack overcloud update prepare command, which updates the overcloud plan to Red Hat OpenStack Platform (RHOSP) 16.2 and prepares the nodes for the update.

Prerequisites

  • If you use a Ceph subscription and have configured director to use the overcloud-minimal image for Ceph storage nodes, you must ensure that in the roles_data.yaml role definition file, the rhsm_enforce parameter is set to False.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update preparation command:

    $ openstack overcloud update prepare \
        --templates \
        --stack <stack_name> \
        -r <roles_data_file> \
        -n <network_data_file> \
        -e <environment_file> \
        -e <environment_file> \
        ...

    Include the following options relevant to your environment:

    • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <stack_name> with the name of your stack.
    • If use your own custom roles, include your custom roles (<roles_data>) file (-r).
    • If use custom networks, include your composable network (<network_data>) file (-n).
    • Any custom configuration environment files (-e).
  3. Wait until the update preparation process completes.

3.2. Running the container image preparation

Before you can update the overcloud, you must prepare all container image configurations that are required for your environment and pull the latest RHOSP 16.2 container images to your undercloud.

To complete the container image preparation, you must run the openstack overcloud external-update run command against tasks that have the container_image_prepare tag.

Note

If you are not using the default stack name, which is overcloud, set your stack name with the --stack <stack_name> option and replace <stack_name> with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the openstack overcloud external-update run command against tasks that have the container_image_prepare tag:

    $ openstack overcloud external-update run --stack <stack_name> --tags container_image_prepare

3.3. Updating all Controller nodes

Update all the Controller nodes to the latest RHOSP 16.2 version. Run the openstack overcloud update run command and include the --limit Controller option to restrict operations to the Controller nodes only. The control plane API is fully available during the minor update.

Important

Until BZ#1872404 is resolved, for nodes based on composable roles, you must update the Database role first, before you can update Controller, Messaging, Compute, Ceph, and other roles.

Note

If you are not using the default stack name, which is overcloud, set your stack name with the --stack <stack_name> option and replace <stack_name> with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --stack <stack_name> --limit Controller --playbook all
  3. Wait until the Controller node update completes.

3.4. Updating all Compute nodes

Update all Compute nodes to the latest RHOSP 16.2 version. To update Compute nodes, run the openstack overcloud update run command and include the --limit Compute option to restrict operations to the Compute nodes only.

Parallelization considerations

When you update a large number of Compute nodes, to improve performance, you can run multiple update tasks in the background and configure each task to update a separate group of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:

$ openstack overcloud update run -y --limit 'Compute[0:19]' > update-compute-0-19.log 2>&1 &
$ openstack overcloud update run -y --limit 'Compute[20:39]' > update-compute-20-39.log 2>&1 &
$ openstack overcloud update run -y --limit 'Compute[40:59]' > update-compute-40-59.log 2>&1 &
$ openstack overcloud update run -y --limit 'Compute[60:79]' > update-compute-60-79.log 2>&1 &

This method of partitioning the nodes space is random and you do not have control over which nodes are updated. The selection of nodes is based on the inventory file that you generate when you run the tripleo-ansible-inventory command.

To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:

$ openstack overcloud update run --limit <Compute0>,<Compute1>,<Compute2>,<Compute3>
Note

If you are not using the default stack name, which is overcloud, set your stack name with the --stack <stack_name> option and replace <stack_name> with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --stack <stack_name> --limit Compute --playbook all
  3. Wait until the Compute node update completes.

3.5. Updating all HCI Compute nodes

Update the Hyperconverged Infrastructure (HCI) Compute nodes to the latest RHOSP 16.2 version. To update the HCI Compute nodes, run the openstack overcloud update run command and include the --limit ComputeHCI option to restrict operations to only the HCI nodes. You must also run the openstack overcloud external-update run --tags ceph command to perform an update to a containerized Red Hat Ceph Storage 4 cluster.

Note

If you are not using the default stack name, which is overcloud, set your stack name with the --stack <stack_name> option and replace <stack_name> with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --stack <stack_name> --limit ComputeHCI --playbook all
  3. Wait until the node update completes.
  4. Run the Ceph Storage update command:

    $ openstack overcloud external-update run --stack <stack_name> --tags ceph
  5. Wait until the Compute HCI node update completes.

3.6. Updating all Ceph Storage nodes

Update the Ceph Storage nodes to the latest RHOSP 16.2 version. To update the Ceph Storage nodes, run the openstack overcloud update run command and include the --limit CephStorage option to restrict operations to only the Ceph Storage nodes only. You must also run the openstack overcloud external-update run command to run ceph-ansible as an external process and update the Red Hat Ceph Storage 4 containers.

Note

If you are not using the default stack name, which is overcloud, set your stack name with the --stack <stack_name> option and replace <stack_name> with the name of your stack.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --stack  <stack_name> --limit CephStorage --playbook all
  3. Wait until the node update completes.
  4. Run the Ceph Storage container update command:

    $ openstack overcloud external-update run --tags ceph
  5. Wait until the Ceph Storage container update completes.

3.7. Performing online database updates

Some overcloud components require an online update or migration of their databases tables. To perform online database updates, run the openstack overcloud external-update run command against tasks that have the online_upgrade tag.

Online database updates apply to the following components:

  • OpenStack Block Storage (cinder)
  • OpenStack Compute (nova)

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the openstack overcloud external-update run command against tasks that use the online_upgrade tag:

    $ openstack overcloud external-update run --tags online_upgrade

3.8. Finalizing the update

To finalize the update to the latest RHOSP 16.2 version, you must update the overcloud stack. This ensures that the stack resource structure aligns with a regular deployment of OSP 16.2 and you can perform standard openstack overcloud deploy functions in the future.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update finalization command:

    $ openstack overcloud update converge \
        --templates \
        --stack <stack_name> \
        -r <roles_data_file> \
        -n <network_data_file> \
        -e <environment_file> \
        -e <environment_file> \
        ...
        ...

    Include the following options that are relevant to your environment:

    • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <stack_name> with the name of your stack.
    • If you use custom roles, include your custom roles (<roles_data>) file (-r).
    • If you use custom networks, include your composable network (<network_data>) file (-n).
    • Any custom configuration environment files (-e).
  3. Wait until the update finalization completes.

Chapter 4. Rebooting the overcloud

After you perform a minor Red Hat OpenStack Platform (RHOSP) update to the latest 16.2 version, reboot your overcloud. The reboot refreshes the nodes with any associated kernel, system-level, and container component updates. These updates provide performance and security benefit. Plan downtime to perform the reboot procedures.

Use the following guidance to understand how to reboot different node types:

4.1. Rebooting Controller and composable nodes

Reboot Controller nodes and standalone nodes based on composable roles, and exclude Compute nodes and Ceph Storage nodes.

Procedure

  1. Log in to the node that you want to reboot.
  2. Optional: If the node uses Pacemaker resources, stop the cluster:

    [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop
  3. Reboot the node:

    [heat-admin@overcloud-controller-0 ~]$ sudo reboot
  4. Wait until the node boots.

Verfication

  1. Verify that the services are enabled.

    1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status
    2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status
    3. If the node uses containerized services, check that all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

4.2. Rebooting a Ceph Storage (OSD) cluster

Complete the following steps to reboot a cluster of Ceph Storage (OSD) nodes.

Procedure

  1. Log into a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

    $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
    $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance
    Note

    If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you set the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name>

  2. Select the first Ceph Storage node that you want to reboot and log in to the node.
  3. Reboot the node:

    $ sudo reboot
  4. Wait until the node boots.
  5. Log into the node and check the cluster status:

    $ sudo podman exec -it ceph-mon-controller-0 ceph status

    Check that the pgmap reports all pgs as normal (active+clean).

  6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes.
  7. When complete, log into a Ceph MON or Controller node and re-enable cluster rebalancing:

    $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
    $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance
    Note

    If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you unset the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name>

  8. Perform a final status check to verify that the cluster reports HEALTH_OK:

    $ sudo podman exec -it ceph-mon-controller-0 ceph status

4.3. Rebooting Compute nodes

To ensure minimal downtime of instances in your Red Hat OpenStack Platform environment, the Migrating instances workflow outlines the steps you must complete to migrate instances from the Compute node that you want to reboot.

Migrating instances workflow

  1. Decide whether to migrate instances to another Compute node before rebooting the node.
  2. Select and disable the Compute node that you want to reboot so that it does not provision new instances.
  3. Migrate the instances to another Compute node.
  4. Reboot the empty Compute node.
  5. Enable the empty Compute node.

Prerequisites

  • Before you reboot the Compute node, you must decide whether to migrate instances to another Compute node while the node is rebooting.

    Review the list of migration constraints that you might encounter when you migrate virtual machine instances between Compute nodes. For more information, see Migration constraints in Configuring the Compute Service for Instance Creation.

  • If you cannot migrate the instances, you can set the following core template parameters to control the state of the instances after the Compute node reboots:

    NovaResumeGuestsStateOnHostBoot
    Determines whether to return instances to the same state on the Compute node after reboot. When set to False, the instances remain down and you must start them manually. The default value is False.
    NovaResumeGuestsShutdownTimeout

    Number of seconds to wait for an instance to shut down before rebooting. It is not recommended to set this value to 0. The default value is 300.

    For more information about overcloud parameters and their usage, see Overcloud Parameters.

    Procedure

    1. Log in to the undercloud as the stack user.
    2. List all Compute nodes and their UUIDs:

      $ source ~/stackrc
      (undercloud) $ openstack server list --name compute

      Identify the UUID of the Compute node that you want to reboot.

    3. From the undercloud, select a Compute node and disable it:

      $ source ~/overcloudrc
      (overcloud) $ openstack compute service list
      (overcloud) $ openstack compute service set <hostname> nova-compute --disable
    4. List all instances on the Compute node:

      (overcloud) $ openstack server list --host <hostname> --all-projects
    5. Optional: If you decide to migrate the instances to another Compute node, complete the following steps:

      1. If you decide to migrate the instances to another Compute node, use one of the following commands:

        • To migrate the instance to a different host, run the following command:

          (overcloud) $ openstack server migrate <instance_id> --live <target_host> --wait
        • Let nova-scheduler automatically select the target host:

          (overcloud) $ nova live-migration <instance_id>
        • Live migrate all instances at once:

          $ nova host-evacuate-live <hostname>
          Note

          The nova command might cause some deprecation warnings, which are safe to ignore.

      2. Wait until migration completes.
      3. Confirm that the migration was successful:

        (overcloud) $ openstack server list --host <hostname> --all-projects
      4. Continue to migrate instances until none remain on the Compute node.
    6. Log in to the Compute node and reboot the node:

      [heat-admin@overcloud-compute-0 ~]$ sudo reboot
    7. Wait until the node boots.
    8. Re-enable the Compute node:

      $ source ~/overcloudrc
      (overcloud) $ openstack compute service set <hostname>  nova-compute --enable
    9. Check that the Compute node is enabled:

      (overcloud) $ openstack compute service list