Chapter 12. Preparing for a director-deployed Ceph Storage upgrade
If your deployment uses a director-deployed Red Hat Ceph Storage cluster, you must complete the procedures included in this section.
If you are upgrading with external Ceph deployments, you must skip the procedures included in this section and continue to Chapter 13, Preparing for upgrading with external Ceph deployments.
The upgrade process maintains the use of Red Hat Ceph Storage 3 containerized services during the upgrade to Red Hat OpenStack Platform 16.1. After you complete the Red Hat OpenStack Platform 16.1 upgrade, you upgrade the Ceph Storage services to Red Hat Ceph Storage 4.
You cannot provision new shares with the Shared File Systems service (manila) until you complete both the Red Hat OpenStack Platform 16.1 upgrade and the Ceph Storage services upgrade to Red Hat Ceph Storage 4.
12.1. Understanding the Ceph Storage node upgrade process at a high level
The director-deployed Ceph Storage nodes continue to use Red Hat Ceph Storage 3 containers during the overcloud upgrade process. To understand how Ceph Storage nodes and services are impacted during the upgrade process, read the following summaries for each aspect of the Ceph Storage upgrade process.
ceph-ansible is a collection of roles and playbooks that director uses to install, maintain, and upgrade Ceph Storage services. When you upgraded the undercloud, you ran certain commands that ensured
ceph-ansible remained at the latest version 3 collection after the transition to Red Hat Enterprise Linux 8.2. Version 3 of
ceph-ansible keeps the containerized Ceph Storage services on version 3 through the duration of the overcloud upgrade. After you complete the upgrade, you enable the Red Hat Ceph Storage update the Red Hat Ceph Storage Tools 4 for RHEL 8 repository and update
ceph-ansible to version 4.
Migration to Podman
During the overcloud upgrade, you must run the
openstack overcloud external-upgrade run --tags ceph_systemd command to change the
systemd services that control Ceph Storage containerized services to use Podman instead of Docker. You run this command before performing the operating system upgrade on any node that contains Ceph Storage containerized services.
After you change the
systemd services to use Podman on a node, you perform the operating system upgrade and the OpenStack Platform service upgrade. The Ceph Storage containers on that node will run again after the OpenStack Platform service upgrade.
Ceph Storage operating system upgrade
You follow the same workflow on Ceph Storage nodes as you do on overcloud nodes in general. When you run the
openstack overcloud upgrade run --tags system_upgrade command against a Ceph Storage node, director runs Leapp on Ceph Storage node and upgrades the operating system to Red Hat Enterprise Linux 8.2. You then run the untagged
openstack overcloud upgrade run command against the Ceph Storage node, which runs the following containers:
- Red Hat Ceph Storage 3 containerized services
- Red Hat OpenStack Platform 16.1 containerized services
Upgrading to Red Hat Ceph Storage 4
After you complete the Leapp upgrade and Red Hat OpenStack Platform upgrade, the Ceph Storage containerized services will still use version 3 containers. At this point, you must upgrade
ceph-ansible to version 4 and then run the
openstack overcloud external-upgrade run --tags ceph command that performs an upgrade of all Red Hat Ceph Storage services on all nodes to version 4.
Summary of the Ceph Storage workflow
The following list is a high level workflow for the Red Hat Ceph Storage upgrade. This workflow is integrated into the general Red Hat OpenStack Platform workflow and you run upgrade framework commands on the undercloud to perform the operations in this workflow.
Upgrade the undercloud but retain version 3 of
- Start the overcloud upgrade
Perform the following tasks for each node that hosts Ceph Storage containerized services:
- Migrate the Ceph Storage containerized services to Podman
- Upgrade the operating system
- Upgrade the OpenStack Platform services, which relaunches Ceph Storage version 3 containerized services
- Complete the overcloud upgrade
ceph-ansibleto version 4 on the undercloud
- Upgrade to Red Hat Ceph Storage 4 on the overcloud
This list does not capture all steps in the complete Red Hat OpenStack Platform 16.1 upgrade process but focuses only on the aspects relevant to Red Hat Ceph Storage to describe what occurs to Ceph Storage services during the upgrade process.
12.2. Checking your ceph-ansible version
During the undercloud upgrade, you retained the Ceph Storage 3 version of the
ceph-ansible package. This helps maintain the compatibility of the Ceph Storage 3 containers on your Ceph Storage nodes. Verify that this package remains on your undercloud.
Log in to the undercloud as the
dnfcommand to check the version of the
$ sudo dnf info ceph-ansible
The command output shows version 3 of the
Installed Packages Name : ceph-ansible Version : 3.xx.xx.xx ...
ceph-ansible package is missing or not a version 3 package, download the latest version 3 package from the Red Hat Package Browser and manually install the package on your undercloud. Note that the
ceph-ansible version 3 package is only available from Red Hat Enterprise Linux 7 repositories and is not available in Red Hat Enterprise Linux 8 repositories.
ceph-ansible version 3 is not supported on Red Hat Enterprise Linux 8 outside the context of the Red Hat OpenStack Platform framework for upgrades.
12.3. Setting the ceph-ansible repository
The Red Hat OpenStack Platform 16.1 validation framework tests that
ceph-ansible is installed correctly before director upgrades the overcloud to Red Hat Ceph Storage 4. The framework uses the
CephAnsibleRepo parameter to check that you installed
ceph-ansible from the correct repository. Director disables the test after you run the
openstack overcloud upgrade prepare command and this test remains disabled through the duration of the Red Hat OpenStack Platform 16.1 overcloud upgrade. Director re-enables this test after running the
openstack overcloud upgrade converge command. However, to prepare for this validation, you must set the
CephAnsibleRepo parameter to the Red Hat Ceph Storage Tools 4 for RHEL 8 repository.
Log in to the undercloud as the
Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named
ceph-config.yamland you can find it in your
$ vi /home/stack/templates/ceph-config.yaml
CephAnsibleRepoparameter to the
parameter_defaults: ... CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms ...
CephAnsibleReposets the repository that includes
ceph-ansible. The validation framework uses this parameter to check that you have installed
ceph-ansibleon the undercloud.
12.4. Checking Ceph cluster status before an upgrade
Before you can proceed with the overcloud upgrade, you must verify that the Ceph cluster is functioning as expected.
Log in to the node that is running the
ceph-monservice. This node is usually a Controller node or a standalone Ceph Monitor node.
Enter the following command to view the status of the Ceph cluster:
$ docker exec ceph-mon-$HOSTNAME ceph -s
Confirm that the health status of the cluster is
HEALTH_OKand that all of the OSDs are