Chapter 12. Preparing for a director-deployed Ceph Storage upgrade
If your deployment uses a director-deployed Red Hat Ceph Storage cluster, you must complete the procedures included in this section.
RHOSP 16.2 is supported on RHEL 8.4. However, hosts that are mapped to the Ceph Storage role update to the latest major RHEL release. For more information, see Red Hat Ceph Storage: Supported configurations.
If you are upgrading with external Ceph deployments, you must skip the procedures included in this section and continue to Chapter 13, Preparing for upgrading with external Ceph deployments.
The upgrade process maintains the use of Red Hat Ceph Storage 3 containerized services during the upgrade to Red Hat OpenStack Platform 16.2. After you complete the Red Hat OpenStack Platform 16.2 upgrade, you upgrade the Ceph Storage services to Red Hat Ceph Storage 4.
You cannot provision new shares with the Shared File Systems service (manila) until you complete both the Red Hat OpenStack Platform 16.2 upgrade and the Ceph Storage services upgrade to Red Hat Ceph Storage 4.
12.1. Understanding the Ceph Storage node upgrade process at a high level
The director-deployed Ceph Storage nodes continue to use Red Hat Ceph Storage 3 containers during the overcloud upgrade process. To understand how Ceph Storage nodes and services are impacted during the upgrade process, read the following summaries for each aspect of the Ceph Storage upgrade process.
ceph-ansible
ceph-ansible
is a collection of roles and playbooks that director uses to install, maintain, and upgrade Ceph Storage services. When you upgraded the undercloud, you ran certain commands that ensured ceph-ansible
remained at the latest version 3 collection after the transition to Red Hat Enterprise Linux 8.4. Version 3 of ceph-ansible
keeps the containerized Ceph Storage services on version 3 through the duration of the overcloud upgrade. After you complete the upgrade, you enable the Red Hat Ceph Storage update the Red Hat Ceph Storage Tools 4 for RHEL 8 repository and update ceph-ansible
to version 4.
Migration to Podman
During the overcloud upgrade, you must run the openstack overcloud external-upgrade run --tags ceph_systemd
command to change the systemd
services that control Ceph Storage containerized services to use Podman instead of Docker. You run this command before performing the operating system upgrade on any node that contains Ceph Storage containerized services.
After you change the systemd
services to use Podman on a node, you perform the operating system upgrade and the OpenStack Platform service upgrade. The Ceph Storage containers on that node will run again after the OpenStack Platform service upgrade.
Ceph Storage operating system upgrade
You follow the same workflow on Ceph Storage nodes as you do on overcloud nodes in general. When you run the openstack overcloud upgrade run --tags system_upgrade
command against a Ceph Storage node, director runs Leapp on Ceph Storage node and upgrades the operating system to Red Hat Enterprise Linux 8.4. You then run the untagged openstack overcloud upgrade run
command against the Ceph Storage node, which runs the following containers:
- Red Hat Ceph Storage 3 containerized services
- Red Hat OpenStack Platform 16.2 containerized services
Upgrading to Red Hat Ceph Storage 4
After you complete the Leapp upgrade and Red Hat OpenStack Platform upgrade, the Ceph Storage containerized services will still use version 3 containers. At this point, you must upgrade ceph-ansible
to version 4 and then run the openstack overcloud external-upgrade run --tags ceph
command that performs an upgrade of all Red Hat Ceph Storage services on all nodes to version 4.
Summary of the Ceph Storage workflow
The following list is a high level workflow for the Red Hat Ceph Storage upgrade. This workflow is integrated into the general Red Hat OpenStack Platform workflow and you run upgrade framework commands on the undercloud to perform the operations in this workflow.
-
Upgrade the undercloud but retain version 3 of
ceph-ansible
- Start the overcloud upgrade
Perform the following tasks for each node that hosts Ceph Storage containerized services:
- Migrate the Ceph Storage containerized services to Podman
- Upgrade the operating system
- Upgrade the OpenStack Platform services, which relaunches Ceph Storage version 3 containerized services
- Complete the overcloud upgrade
-
Upgrade
ceph-ansible
to version 4 on the undercloud - Upgrade to Red Hat Ceph Storage 4 on the overcloud
This list does not capture all steps in the complete Red Hat OpenStack Platform 16.2 upgrade process but focuses only on the aspects relevant to Red Hat Ceph Storage to describe what occurs to Ceph Storage services during the upgrade process.
12.2. Checking your ceph-ansible version
During the undercloud upgrade, you retained the Ceph Storage 3 version of the ceph-ansible
package. This helps maintain the compatibility of the Ceph Storage 3 containers on your Ceph Storage nodes. Verify that this package remains on your undercloud.
Procedure
-
Log in to the undercloud as the
stack
user. Run the
dnf
command to check the version of theceph-ansible
package:$ sudo dnf info ceph-ansible
The command output shows version 3 of the
ceph-ansible
package:Installed Packages Name : ceph-ansible Version : 3.xx.xx.xx ...
If the ceph-ansible
package is missing or not a version 3 package, download the latest version 3 package from the Red Hat Package Browser and manually install the package on your undercloud. Note that the ceph-ansible
version 3 package is only available from Red Hat Enterprise Linux 7 repositories and is not available in Red Hat Enterprise Linux 8 repositories. ceph-ansible
version 3 is not supported on Red Hat Enterprise Linux 8 outside the context of the Red Hat OpenStack Platform framework for upgrades.
12.3. Setting the ceph-ansible repository
The Red Hat OpenStack Platform 16.2 validation framework tests that ceph-ansible
is installed correctly before director upgrades the overcloud to Red Hat Ceph Storage 4. The framework uses the CephAnsibleRepo
parameter to check that you installed ceph-ansible
from the correct repository. Director disables the test after you run the openstack overcloud upgrade prepare
command and this test remains disabled through the duration of the Red Hat OpenStack Platform 16.2 overcloud upgrade. Director re-enables this test after running the openstack overcloud upgrade converge
command. However, to prepare for this validation, you must set the CephAnsibleRepo
parameter to the Red Hat Ceph Storage Tools 4 for RHEL 8 repository.
Procedure
-
Log in to the undercloud as the
stack
user. Edit the environment file that contains your overcloud Ceph Storage configuration. This file is usually named
ceph-config.yaml
and you can find it in yourtemplates
directory:$ vi /home/stack/templates/ceph-config.yaml
Add the
CephAnsibleRepo
parameter to theparameter_defaults
section:parameter_defaults: ... CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms ...
CephAnsibleRepo
sets the repository that includesceph-ansible
. The validation framework uses this parameter to check that you have installedceph-ansible
on the undercloud.-
Save the
ceph-config.yaml
file.
12.4. Checking Ceph cluster status before an upgrade
Before you can proceed with the overcloud upgrade, you must verify that the Ceph cluster is functioning as expected.
Procedure
-
Log in to the node that is running the
ceph-mon
service. This node is usually a Controller node or a standalone Ceph Monitor node. Enter the following command to view the status of the Ceph cluster:
$ docker exec ceph-mon-$HOSTNAME ceph -s
-
Confirm that the health status of the cluster is
HEALTH_OK
and that all of the OSDs areup
.