Chapter 12. Preparing for a director-deployed Ceph Storage upgrade

If your deployment uses a director-deployed Red Hat Ceph Storage cluster, you must complete the procedures included in this section.

Note

If you are upgrading with external Ceph deployments, you must skip the procedures included in this section and continue to Chapter 13, Preparing for upgrading with external Ceph deployments.

The upgrade process maintains the use of Red Hat Ceph Storage 3 containerized services during the upgrade to Red Hat OpenStack Platform 16.1. After you complete the Red Hat OpenStack Platform 16.1 upgrade, you upgrade the Ceph Storage services to Red Hat Ceph Storage 4.

You cannot provision new shares with the Shared File Systems service (manila) until you complete both the Red Hat OpenStack Platform 16.1 upgrade and the Ceph Storage services upgrade to Red Hat Ceph Storage 4.

12.1. Understanding the Ceph Storage node upgrade process at a high level

The director-deployed Ceph Storage nodes continue to use Red Hat Ceph Storage 3 containers during the overcloud upgrade process. To understand how Ceph Storage nodes and services are impacted during the upgrade process, read the following summaries for each aspect of the Ceph Storage upgrade process.

ceph-ansible

ceph-ansible is a collection of roles and playbooks that director uses to install, maintain, and upgrade Ceph Storage services. When you upgraded the undercloud, you ran certain commands that ensured ceph-ansible remained at the latest version 3 collection after the transition to Red Hat Enterprise Linux 8.2. Version 3 of ceph-ansible keeps the containerized Ceph Storage services on version 3 through the duration of the overcloud upgrade. After you complete the upgrade, you enable the Red Hat Ceph Storage update the Red Hat Ceph Storage Tools 4 for RHEL 8 repository and update ceph-ansible to version 4.

Migration to Podman

During the overcloud upgrade, you must run the openstack overcloud external-upgrade run --tags ceph_systemd command to change the systemd services that control Ceph Storage containerized services to use Podman instead of Docker. You run this command before performing the operating system upgrade on any node that contains Ceph Storage containerized services.

After you change the systemd services to use Podman on a node, you perform the operating system upgrade and the OpenStack Platform service upgrade. The Ceph Storage containers on that node will run again after the OpenStack Platform service upgrade.

Ceph Storage operating system upgrade

You follow the same workflow on Ceph Storage nodes as you do on overcloud nodes in general. When you run the openstack overcloud upgrade run --tags system_upgrade command against a Ceph Storage node, director runs Leapp on Ceph Storage node and upgrades the operating system to Red Hat Enterprise Linux 8.2. You then run the untagged openstack overcloud upgrade run command against the Ceph Storage node, which runs the following containers:

  • Red Hat Ceph Storage 3 containerized services
  • Red Hat OpenStack Platform 16.1 containerized services

Upgrading to Red Hat Ceph Storage 4

After you complete the Leapp upgrade and Red Hat OpenStack Platform upgrade, the Ceph Storage containerized services will still use version 3 containers. At this point, you must upgrade ceph-ansible to version 4 and then run the openstack overcloud external-upgrade run --tags ceph command that performs an upgrade of all Red Hat Ceph Storage services on all nodes to version 4.

Summary of the Ceph Storage workflow

The following list is a high level workflow for the Red Hat Ceph Storage upgrade. This workflow is integrated into the general Red Hat OpenStack Platform workflow and you run upgrade framework commands on the undercloud to perform the operations in this workflow.

  1. Upgrade the undercloud but retain version 3 of ceph-ansible
  2. Start the overcloud upgrade
  3. Perform the following tasks for each node that hosts Ceph Storage containerized services:

    1. Migrate the Ceph Storage containerized services to Podman
    2. Upgrade the operating system
    3. Upgrade the OpenStack Platform services, which relaunches Ceph Storage version 3 containerized services
  4. Complete the overcloud upgrade
  5. Upgrade ceph-ansible to version 4 on the undercloud
  6. Upgrade to Red Hat Ceph Storage 4 on the overcloud
Note

This list does not capture all steps in the complete Red Hat OpenStack Platform 16.1 upgrade process but focuses only on the aspects relevant to Red Hat Ceph Storage to describe what occurs to Ceph Storage services during the upgrade process.

12.2. Checking your ceph-ansible version

During the undercloud upgrade, you retained the Ceph Storage 3 version of the ceph-ansible package. This helps maintain the compatibility of the Ceph Storage 3 containers on your Ceph Storage nodes. Verify that this package remains on your undercloud.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Run the dnf command to check the version of the ceph-ansible package:

    $ sudo dnf info ceph-ansible

    The command output shows version 3 of the ceph-ansible package:

    Installed Packages
    Name         : ceph-ansible
    Version      : 3.xx.xx.xx
    ...
Important

If the ceph-ansible package is missing or not a version 3 package, download the latest version 3 package from the Red Hat Package Browser and manually install the package on your undercloud. Note that the ceph-ansible version 3 package is only available from Red Hat Enterprise Linux 7 repositories and is not available in Red Hat Enterprise Linux 8 repositories. ceph-ansible version 3 is not supported on Red Hat Enterprise Linux 8 outside the context of the Red Hat OpenStack Platform framework for upgrades.

12.3. Setting the ceph-ansible repository

The Red Hat OpenStack Platform 16.1 validation framework tests that ceph-ansible is installed correctly before director upgrades the overcloud to Red Hat Ceph Storage 4. The framework uses the CephAnsibleRepo parameter to check that you installed ceph-ansible from the correct repository. Director disables the test after you run the openstack overcloud upgrade prepare command and this test remains disabled through the duration of the Red Hat OpenStack Platform 16.1 overcloud upgrade. Director re-enables this test after running the openstack overcloud upgrade converge command. However, to prepare for this validation, you must set the CephAnsibleRepo parameter to the Red Hat Ceph Storage Tools 4 for RHEL 8 repository.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named ceph-config.yaml and you can find it in your templates directory:

    $ vi /home/stack/templates/ceph-config.yaml
  3. Add the CephAnsibleRepo parameter to the parameter_defaults section:

    parameter_defaults:
      ...
      CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms
      ...

    CephAnsibleRepo sets the repository that includes ceph-ansible. The validation framework uses this parameter to check that you have installed ceph-ansible on the undercloud.

  4. Save the ceph-config.yaml file.

12.4. Checking Ceph cluster status before an upgrade

Before you can proceed with the overcloud upgrade, you must verify that the Ceph cluster is functioning as expected.

Procedure

  1. Log in to the node that is running the ceph-mon service. This node is usually a Controller node or a standalone Ceph Monitor node.
  2. Enter the following command to view the status of the Ceph cluster:

    $ docker exec ceph-mon-$HOSTNAME ceph -s
  3. Confirm that the health status of the cluster is HEALTH_OK and that all of the OSDs are up.