Chapter 4. Upgrading a Red Hat Ceph Storage cluster from RHCS 4 to RHCS 5

As a storage administrator, you can upgrade a Red Hat Ceph Storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5. The upgrade process includes the following tasks:

  • Upgrade the host OS version on the storage cluster from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8, if your storage cluster is still running Red Hat Enterprise Linux 7.
  • Use Ansible playbooks to upgrade a Red Hat Ceph Storage 4 storage cluster to Red Hat Ceph Storage 5.
Important

ceph-ansible is currently not supported with Red Hat Ceph Storage 5. This means that once you have migrated your storage cluster to Red Hat Ceph Storage 5, you must use cephadm and cephadm-ansible to perform subsequent updates.

Important

Upgrading a storage cluster with an RGW NFS gateway from Red Hat Ceph Storage 4 to to Red Hat Ceph Storage 5 is currently not supported. The ceph-ansible upgrade fails and returns an error message. RGW NFS gateway support will be included in a later version of Red Hat Ceph Storage 5.

Important

The option bluefs_buffered_io is set to True by default for Red Hat Ceph Storage. This option enables BlueFS to perform buffered reads in some cases, and enables the kernel page cache to act as a secondary cache for reads like RocksDB block reads. For example, if the RocksDB block cache is not large enough to hold all blocks during the OMAP iteration, it may be possible to read them from the page cache instead of the disk. This can dramatically improve performance when osd_memory_target is too small to hold all entries in the block cache. Currently, enabling bluefs_buffered_io and disabling the system level swap prevents performance degradation.

Red Hat Ceph Storage 5 supports only containerized daemons. It does not support non-containerized storage clusters. If you are upgrading a non-containerized storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, the upgrade process includes the conversion to a containerized deployment.

4.1. Prerequisites

  • A running Red Hat Ceph Storage 4 cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage tools and Ansible repositories are enabled.
Important

You can manually upgrade the Ceph File System (CephFS) Metadata Server (MDS) software on a Red Hat Ceph Storage cluster and the Red Hat Enterprise Linux operating system to a new major release at the same time. The underlying XFS filesystem must be formatted with ftype=1 or with d_type support. Run the command xfs_info /var to ensure the ftype is set to 1. If the value of ftype is not 1, attach a new disk or create a volume. On top of this new device, create a new XFS filesystem and mount it on /var/lib/containers.

Starting with Red Hat Enterprise Linux 8, mkfs.xfs enables ftype=1 by default.

4.2. Compatibility considerations between RHCS and podman versions

podman and Red Hat Ceph Storage have different end-of-life strategies that might make it challenging to find compatible versions.

If you plan to upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 as part of the Ceph upgrade process, make sure that the version of podman is compatible with Red Hat Ceph Storage 5.0.

Important

Red Hat Ceph Storage 5.0 is compatible with podman versions 2.0.0 and later, except for version 2.2.1. Version 2.2.1 is not compatible with Red Hat Ceph Storage 5.0.

The following table shows version compatibility between Red Hat Ceph Storage 5.0 and versions of podman.

CephPodman    
 

1.9

2.0

2.1

2.2

3.0

5.0 (Pacific)

false

true

true

false

true

4.3. Preparing for an upgrade

As a storage administrator, you can upgrade your Ceph storage cluster to Red Hat Ceph Storage 5. However, some components of your storage cluster must be running specific software versions before an upgrade can take place. The following list shows the minimum software versions that must be installed on your storage cluster before you can upgrade to Red Hat Ceph Storage 5.

  • RHCS 4.2z2 or later.
  • Ansible 2.9.
  • Ceph-ansible shipped with the latest version of RHCS.
  • RHEL 8.4.
  • FileStore OSDs must be migrated to BlueStore. For more information about converting OSDs from FileStore to BlueStore, refer to BlueStore.

There is no direct upgrade path from RHCS versions earlier than RHCS 4.2z2. If you are upgrading from RHCS 3, you must first upgrade to RHCS 4.2z2 or later, and then upgrade to RHCS 5.

Important

You can only upgrade to the latest version of Red Hat Ceph Storage 5. For example, if version 5.1 is available, you cannot upgrade from 4 to 5.0; you must go directly to 5.1.

To upgrade your storage cluster to RHCS 5, Red Hat recommends that your cluster be running RHCS 4.2z2 or later. Refer to the Knowledgebase article What are the Red Hat Ceph Storage Releases?. This article contains download links to the most recent versions of the Ceph packages and ceph-ansible.

The upgrade process uses Ansible playbooks to upgrade an Red Hat Ceph Storage 4 storage cluster to Red Hat Ceph Storage 5. If your Red Hat Ceph Storage 4 cluster is a non-containerized cluster, the upgrade process includes a step to transform the cluster into a containerized version. Red Hat Ceph Storage 5 does not run on non-containerized clusters.

If you have a mirroring or multisite configuration, upgrade one cluster at a time. Make sure that each upgraded cluster is running properly before upgrading another cluster.

Important

leapp does not support upgrades for encrypted OSDs or OSDs that have encrypted partitions. If your OSDs are encrypted and you are upgrading the host OS, disable dmcrypt in ceph-ansible before upgrading the OS. For more information about using leapp, refer to Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.

Important

Perform the first three steps in this procedure only if the storage cluster is not already running the latest version of RHCS 4. The latest version of RHCS 4 should be 4.2z2 or later.

Prerequisites

  • A running Red Hat Ceph Storage 4 cluster.
  • Sudo-level access to all nodes in the storage cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage tools and Ansible repositories are enabled.

Procedure

  1. Enable the Ceph and Ansible repositories on the Ansible administration node:

    Example

    [root@admin ceph-ansible]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

  2. Use the --extra-vars option to update the infrastructure-playbooks/rolling_update.yml playbook and to change the health_osd_check_retries and health_osd_check_delay values to 50 and 30, respectively:

    Example

    [root@admin ceph-ansible]# ansible-playbook -i hosts infrastructure-playbooks/rolling_update.yml --extra-vars "health_osd_check_retries=50 health_osd_check_delay=30"

    For each OSD node, these values cause ceph-ansible to check the storage cluster health every 30 seconds, up to 50 times. This means that ceph-ansible waits up to 25 minutes for each OSD.

    Adjust the health_osd_check_retries option value up or down, based on the used storage capacity of the storage cluster. For example, if you are using 218 TB out of 436 TB, or 50% of the storage capacity, then set the health_osd_check_retries option to 50.

    /etc/ansible/hosts is the default location for the Ansible inventory file.

  3. If the storage cluster you want to upgrade contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to create a denylist for clients:

    Syntax

    ceph auth caps client.ID mon 'allow r, allow command "osd blacklist"' osd 'EXISTING_OSD_USER_CAPS'

  4. If the storage cluster was originally installed using Cockpit, create a symbolic link in the /usr/share/ceph-ansible directory to the inventory file where Cockpit created it, at /usr/share/ansible-runner-service/inventory/hosts:

    1. Change to the /usr/share/ceph-ansible directory:

      # cd /usr/share/ceph-ansible
    2. Create the symbolic link:

      # ln -s /usr/share/ansible-runner-service/inventory/hosts hosts
  5. To upgrade the cluster using ceph-ansible, create the symbolic link in the etc/ansible/hosts directory to the hosts inventory file:

    # ln -s /etc/ansible/hosts hosts
  6. If the storage cluster was originally installed using Cockpit, copy the Cockpit-generated SSH keys to the Ansible user’s ~/.ssh directory:

    1. Copy the keys:

      Syntax

      cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      cp /usr/share/ansible-runner-service/env/ssh_key /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Replace ANSIBLE_USERNAME with the user name for Ansible. The usual default user name is admin.

      Example

      # cp /usr/share/ansible-runner-service/env/ssh_key.pub /home/admin/.ssh/id_rsa.pub
      # cp /usr/share/ansible-runner-service/env/ssh_key /home/admin/.ssh/id_rsa

    2. Set the appropriate owner, group, and permissions on the key files:

      Syntax

      # chown ANSIBLE_USERNAME:_ANSIBLE_USERNAME_ /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chown ANSIBLE_USERNAME:_ANSIBLE_USERNAME_ /home/ANSIBLE_USERNAME/.ssh/id_rsa
      # chmod 644 /home/ANSIBLE_USERNAME/.ssh/id_rsa.pub
      # chmod 600 /home/ANSIBLE_USERNAME/.ssh/id_rsa

      Replace ANSIBLE_USERNAME with the username for Ansible. The usual default user name is admin.

      Example

      # chown admin:admin /home/admin/.ssh/id_rsa.pub
      # chown admin:admin /home/admin/.ssh/id_rsa
      # chmod 644 /home/admin/.ssh/id_rsa.pub
      # chmod 600 /home/admin/.ssh/id_rsa

Additional Resources

4.4. Backing up the files before the host OS upgrade

Note

Perform the procedure in this section only if you are upgrading the host OS. If you are not upgrading the host OS, skip this section.

Before you can perform the upgrade procedure, you must make backup copies of the files that you customized for your storage cluster, including keyring files and the yml files for your configuration.

Prerequisites

  • A running Red Hat Ceph Storage 4 cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage Tools and Ansible repositories are enabled.

Procedure

  1. Make a backup copy of the ceph.client.admin.keyring file.
  2. Make backup copies of the ceph.conf files from each node.
  3. Make backup copies of the /etc/ganesha/ folder on each node.
  4. If the storage cluster has RBD mirroring defined, then make backup copies of the /etc/ceph folder and the group_vars/rbdmirrors.yml file.

Additional Resources

4.5. Converting to a containerized deployment

This procedure is required for non-containerized clusters. If your storage cluster is a non-containerized cluster, this procedure transforms the cluster into a containerized version.

Red Hat Ceph Storage 5 supports container-based deployments only.

If your Red Hat Ceph Storage 4 storage cluster is already containerized, skip this section.

Important

This procedure stops and restarts a daemon. If the playbook stops executing during this procedure, be sure to analyze the state of the cluster before restarting.

Prerequisites

  • A running Red Hat Ceph Storage 4 cluster.
  • Root-level access to all nodes in the storage cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage tools and Ansible repositories are enabled.

Procedure

  1. If you are running a multisite setup, set rgw_multisite: false in all.yml.
  2. Ensure the group_vars/all.yml has the following default values for the configuration parameters:

    ceph_docker_image_tag: "latest"
    ceph_docker_registry: "registry.redhat.io"
    ceph_docker_image: rhceph/rhceph-4-rhel8
    containerized_deployment: true
    Note

    These values differ if you use a local registry and a custom image name.

  3. If you are using daemons that are not containerized, convert them to containerized format:

    Syntax

    ansible-playbook -vvvv -i INVENTORY-FILE infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

    The -vvvv option collects verbose logs of the conversion process.

    Example

    [ansible@admin ceph-ansible]$ ansible-playbook -vvvv -i hosts infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

  4. Once the playbook completes successfully, edit the the value of rgw_multisite: true`in the `all.yml file and ensure the value of containerized_deployment is true.

4.6. Updating the host operating system

Red Hat Ceph Storage 5 supports Red Hat Enterprise Linux 8.4 and later. This procedure enables you to install Red Hat Ceph Storage 5 and Red Hat Enterprise Linux 8 on the nodes in the storage cluster. If you are already running Red Hat Enterprise Linux 8 on your storage cluster, skip this procedure.

You must manually upgrade all other nodes in the cluster to run the most recent versions of Red Hat Enterprise Linux and Red Hat Ceph Storage.

Prerequisites

  • A running Red Hat Ceph Storage 4 storage cluster.
  • Sudo-level access to all nodes in the storage cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage tools and Ansible repositories are enabled.

Procedure

  1. Use the docker-to-podman playbook to convert docker to podman:

    Example

    [ansible@admin ceph-ansible]$ ansible-playbook -vvvv -i hosts infrastructure-playbooks/
    docker-to-podman.yml

4.6.1. Manually upgrading Ceph Monitor nodes and their operating systems

As a system administrator, you can manually upgrade the Ceph Monitor software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Important

Perform the procedure on only one Monitor node at a time. To prevent cluster access issues, ensure that the current upgraded Monitor node has returned to normal operation before proceeding to the next node.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The nodes are running Red Hat Enterprise Linux 7.8.
  • The nodes are using Red Hat Ceph Storage version 4.2z2 or later.
  • Access to the installation source is available for Red Hat Enterprise Linux 8.4.

Procedure

  1. Stop the monitor service:

    Syntax

    systemctl stop ceph-mon@MONITOR_ID

    Replace MONITOR_ID with the Monitor node’s ID number.

  2. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories.

    1. Disable the tools repository:

      # subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms
    2. Disable the mon repository:

      # subscription-manager repos --disable=rhel-7-server-rhceph-4-mon-rpms
  3. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  4. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
  5. Set PermitRootLogin yes in /etc/ssh/sshd_config.
  6. Restart the OpenSSH SSH daemon:

    # systemctl restart sshd.service
  7. Remove the iSCSI module from the Linux kernel:

    # modprobe -r iscsi
  8. Reboot the node.
  9. Enable the repositories for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8.

    1. Enable the tools repository:

      # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
  10. Restore the ceph-client-admin.keyring and ceph.conf files from a Monitor node which has not been upgraded yet or from a node that has already had those files restored.
  11. Verify that the monitor and manager services came back up and that the monitor is in quorum.

    Syntax

    ceph -s

    On the mon: line under services:, ensure that the node is listed as in quorum and not as out of quorum.

    Example

    # ceph -s
    mon: 3 daemons, quorum jb-ceph4-mon,jb-ceph4-mon2,jb-ceph4-mon3 (age 2h)
    mgr: jb-ceph4-mon(active, since 2h), standbys: jb-ceph4-mon3, jb-ceph4-mon2

  12. Repeat the above steps on all Monitor nodes until they have all been upgraded.

Additional Resources

4.6.2. Upgrading the OSD nodes

As a system administrator, you can manually upgrade the Ceph OSD software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Important

Perform this procedure for each OSD node in the Ceph cluster, but typically only for one OSD node at a time. A maximum of one failure domain’s worth of OSD nodes may be performed in parallel. For example, if per-rack replication is in use, one entire rack’s OSD nodes can be upgraded in parallel. To prevent data access issues, ensure that the OSDs of the current OSD node have returned to normal operation and that all of the cluster PGs are in the active+clean state before proceeding to the next OSD.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The nodes are running Red Hat Enterprise Linux 7.9.
  • The nodes are using Red Hat Ceph Storage version 4.2z2 or later.
  • Access to the installation source for Red Hat Enterprise Linux 8.4 or later.
  • FileStore OSDs must be migrated to BlueStore.

Procedure

  1. If you have FileStore OSDs that have not been migrated to BlueStore, run the filestore-to-bluestore playbook. For more information about converting OSDs from FileStore to BlueStore, refer to BlueStore.
  2. Set the OSD noout flag to prevent OSDs from getting marked down during the migration:

    Syntax

    ceph osd set noout

  3. Set the OSD nobackfill, norecover, norrebalance, noscrub and nodeep-scrub flags to avoid unnecessary load on the cluster and to avoid any data reshuffling when the node goes down for migration:

    Syntax

    ceph osd set nobackfill
    ceph osd set norecover
    ceph osd set norebalance
    ceph osd set noscrub
    ceph osd set nodeep-scrub

  4. Gracefully shut down all the OSD processes on the node:

    Syntax

    systemctl stop ceph-osd.target

  5. If using Red Hat Ceph Storage 4, disable the Red Hat Ceph Storage 4 repositories.

    1. Disable the tools repository:

      Syntax

      subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms

    2. Disable the osd repository:

      Syntax

      # subscription-manager repos --disable=rhel-7-server-rhceph-4-osd-rpms

  6. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  7. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
  8. Set PermitRootLogin yes in /etc/ssh/sshd_config.
  9. Restart the OpenSSH SSH daemon:

    Syntax

    systemctl restart sshd.service

  10. Remove the iSCSI module from the Linux kernel:

    Syntax

    modprobe -r iscsi

  11. Perform the upgrade by following Performing the upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.

    1. Enable the tools repository:

      # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
  12. Restore the ceph.conf file.
  13. Unset the noout, nobackfill, norecover, norebalance, noscrub and nodeep-scrub flags:

    Syntax

    ceph osd unset noout
    ceph osd unset nobackfill
    ceph osd unset norecover
    ceph osd unset norebalance
    ceph osd unset noscrub
    ceph osd unset nodeep-scrub

  14. Verify that the OSDs are up and in, and that they are in the active+clean state.

    Syntax

    ceph -s

    On the osd: line under services:, ensure that all OSDs are up and in:

    Example

    # ceph-s
    osd: 3 osds: 3 up (since 8s), 3 in (since 3M)

  15. Repeat this procedure on all OSD nodes until they have all been upgraded.

Additional Resources

4.6.3. Upgrading the Ceph Object Gateway nodes

As a system administrator, you can manually upgrade the Ceph Object Gateway (RGW) software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Important

Perform this procedure for each RGW node in the Ceph cluster, but only for one RGW node at a time. To prevent client access issues, ensure that the current upgraded RGW has returned to normal operation before proceeding to upgrade the next node.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The nodes are running Red Hat Enterprise Linux 7.8 or later
  • The nodes are using Red Hat Ceph Storage version 4.2z2 or later.
  • Access to the installation source for Red Hat Enterprise Linux 8.4 or later.

Procedure

  1. Stop the Ceph Object Gateway service:

    Syntax

    # systemctl stop ceph-radosgw.target

  2. Disable the Red Hat Ceph Storage 4 tools repository:

    # subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms
  3. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  4. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
  5. Set PermitRootLogin yes in /etc/ssh/sshd_config.
  6. Restart the OpenSSH SSH daemon:

    # systemctl restart sshd.service
  7. Remove the iSCSI module from the Linux kernel:

    # modprobe -r iscsi
  8. Perform the upgrade by following Performing the upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  9. Enable the tools repository:

    Syntax

    subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

  10. Restore the ceph-client-admin.keyring and ceph.conf files.
  11. Verify that the daemon is active:

    Syntax

    ceph -s

    View the rgw: line under services: to make sure that the RGW daemon is active.

    Example

    rgw: 1 daemon active (jb-ceph4-rgw.rgw0)

  12. Repeat the above steps on all Ceph Object Gateway nodes until they have all been upgraded.

Additional Resources

4.6.4. Upgrading the CephFS Metadata Server nodes

As a storage administrator, you can manually upgrade the Ceph File System (CephFS) Metadata Server (MDS) software on a Red Hat Ceph Storage cluster and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Important

Before you upgrade the storage cluster, reduce the number of active MDS ranks to one per file system. This eliminates any possible version conflicts between multiple MDS. In addition, take all standby nodes offline before upgrading.

This is because the MDS cluster does not possess built-in versioning or file system flags. Without these features, multiple MDS might communicate using different versions of the MDS software, and could cause assertions or other faults to occur.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The nodes are running Red Hat Enterprise Linux 7.8 or later.
  • The nodes are using Red Hat Ceph Storage version 4.2z2 or later.
  • Access to the installation source for Red Hat Enterprise Linux 8.4 or later.
  • Root-level access to all nodes in the storage cluster.

Procedure

  1. Reduce the number of active MDS ranks to 1:

    Syntax

    ceph fs set FILE_SYSTEM_NAME max_mds 1

    Example

    [root@mds ~]# ceph fs set fs1 max_mds 1

  2. Wait for the cluster to stop all of the MDS ranks. When all of the MDS have stopped, only rank 0 should be active. The rest should be in standby mode. Check the status of the file system:

    [root@mds ~]# ceph status
  3. Use systemctl to take all standby MDS offline:

    [root@mds ~]# systemctl stop ceph-mds.target
  4. Confirm that only one MDS is online, and that it has rank 0 for the file system:

    [root@mds ~]# ceph status
  5. Disable the Red Hat Ceph Storage 4 tools repository:

    [root@mds ~]# subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms
  6. Install the leapp utility. For more information about leapp, refer to Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  7. Run through the leapp preupgrade checks. For more information, refer to Assessing upgradability from the command line.
  8. Edit /etc/ssh/sshd_config and set PermitRootLogin to yes.
  9. Restart the OpenSSH SSH daemon:

    [root@mds ~]# systemctl restart sshd.service
  10. Remove the iSCSI module from the Linux kernel:

    [root@mds ~]# modprobe -r iscsi
  11. Perform the upgrade. See Performing the upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  12. Enable the tools repository:

    Syntax

    subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

  13. Restore the ceph-client-admin.keyring and ceph.conf files.
  14. Verify that the daemon is active:

    [root@mds ~]# ceph -s
  15. Follow the same processes for the standby daemons.
  16. When you have finished restarting all of the MDS in standby, restore the previous value of max_mds for your cluster:

    Syntax

    ceph fs set FILE_SYSTEM_NAME max_mds ORIGINAL_VALUE

    Example

    [root@mds ~]# ceph fs set fs1 max_mds 5

Additional Resources

4.6.5. Manually upgrading the Ceph Dashboard node and its operating system

As a system administrator, you can manually upgrade the Ceph Dashboard software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The node is running Red Hat Enterprise Linux 7.
  • The node is running Red Hat Ceph Storage version 4.2z2 or later.
  • Access is available to the installation source for Red Hat Enterprise Linux 8.4.

Procedure

  1. Disable the Red Hat Ceph Storage 4 tools repository:

    # subscription-manager repos --disable=rhel-7-server-rhceph-4-tools-rpms
  2. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  3. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
  4. Set PermitRootLogin yes in /etc/ssh/sshd_config.
  5. Restart the OpenSSH SSH daemon:

    # systemctl restart sshd.service
  6. Remove the iSCSI module from the Linux kernel:

    # modprobe -r iscsi
  7. Perform the upgrade by following Performing the upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  8. Enable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8:

    # subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

4.6.6. Manually upgrading Ceph Ansible nodes and reconfiguring settings

Manually upgrade the Ceph Ansible software on a Red Hat Ceph Storage cluster node and the Red Hat Enterprise Linux operating system to a new major release at the same time.

Important

Before upgrading the host OS on the Ceph Ansible nodes, back up the group_vars and hosts files. Use the created backups before reconfiguring the Ceph Ansible nodes.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The node is running Red Hat Enterprise Linux 7.
  • The node is running Red Hat Ceph Storage version 4.2z2 or later.
  • Access is available to the installation source for Red Hat Enterprise Linux 8.4.

Procedure

  1. Disable the tools repository for Red Hat Ceph Storage 4 for Red Hat Enterprise Linux 8:

    [root@ansible ~]# subscription-manager repos --disable=rhceph-4-tools-for-rhel-8-x86_64-rpms
    [root@ansible ~]# subscription-manager repos --disable=ansible-2.9-for-rhel-8-x86_64-rpms
  2. Install the leapp utility. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.
  3. Run through the leapp preupgrade checks. See Assessing upgradability from the command line.
  4. Edit /etc/ssh/sshd_config and set PermitRootLogin to yes.
  5. Restart the OpenSSH SSH daemon:

    [root@mds ~]# systemctl restart sshd.service
  6. Remove the iSCSI module from the Linux kernel:

    [root@mds ~]# modprobe -r iscsi
  7. Perform the upgrade. See Upgrading from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.

    Syntax

    subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

  8. Restore the ceph-client-admin.keyring and ceph.conf files.

Additional Resources

4.7. Restoring the backup files

After you have completed the host OS upgrade on each node in your storage cluster, restore all the files that you backed up earlier to each node so that your upgraded node uses your preserved settings.

Repeat this process on each host in your storage cluster after the OS upgrade process for that host is complete.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all nodes in the storage cluster.

Procedure

  1. Restore the files that you backed up before the host OS upgrade to the host.
  2. Restore the /etc/ceph folders and their contents to all of the hosts, including the ceph.client.admin.keyring and ceph.conf files.
  3. Restore the /etc/ganesha/ folder to each node.
  4. Check to make sure that the ownership for each of the backed-up files has not changed after the operating system upgrade. The file owner should be ceph. If the file owner has been changed to root, use the following command on each file to change the ownership back to ceph:

    Example

    [root@admin]# chown ceph: ceph.client.rbd-mirror.node01.keyring

  5. If you upgraded from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8 and the storage cluster had RBD mirroring defined, restore the /etc/ceph folder from the backup copy.
  6. Restore the group_vars/rbdmirrors.yml file that you backed up earlier.

4.8. Backing up the files before the RHCS upgrade

Before you run the rolling_update.yml playbook to upgrade Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, make backup copies of all the yml files.

Prerequisites

  • A Red Hat Ceph Storage 4 cluster running RHCS 4.2z2 or later.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The Ansible user account for use with the Ansible application.
  • Red Hat Ceph Storage tools and Ansible repositories are enabled.

Procedure

  • Make backup copies of all the yml files.

    Example

    [root@admin ceph-ansible]# cp group_vars/all.yml group_vars/all_old.yml
    [root@admin ceph-ansible]# cp group_vars/osds.yml group_vars/osds_old.yml
    [root@admin ceph-ansible]# cp group_vars/mdss.yml group_vars/mdss_old.yml
    [root@admin ceph-ansible]# cp group_vars/rgws.yml group_vars/rgws_old.yml
    [root@admin ceph-ansible]# cp group_vars/clients.yml group_vars/clients_old.yml

4.9. The upgrade process

As a storage administrator, you use Ansible playbooks to upgrade an Red Hat Ceph Storage 4 storage cluster to Red Hat Ceph Storage 5. The rolling_update.yml Ansible playbook performs upgrades for deployments of Red Hat Ceph Storage. The ceph-ansible upgrades the Ceph nodes in the following order:

  • Ceph Monitor
  • Ceph Manager
  • Ceph OSD nodes
  • MDS nodes
  • Ceph Object Gateway (RGW) nodes
  • Ceph RBD-mirror node
  • Ceph NFS nodes
  • Ceph iSCSI gateway node
  • Ceph client nodes
  • Ceph-crash daemons
  • Node-exporter on all nodes
  • Ceph Dashboard
Note

Red Hat Ceph Storage 5 supports only containerized deployments.

ceph-ansible is currently not supported with Red Hat Ceph Storage 5. This means that once you have migrated your storage cluster to Red Hat Ceph Storage 5, you must use cephadm to perform subsequent updates.

Note

Red Hat Ceph Storage 5 also includes a health check function that returns a DAEMON_OLD_VERSION warning if it detects that any of the daemons in the storage cluster are running multiple versions of Red Hat Ceph Storage. The warning is triggered when the daemons continue to run multiple versions of Red Hat Ceph Storage beyond the time value set in the mon_warn_older_version_delay option. By default, the mon_warn_older_version_delay option is set to one week. This setting allows most upgrades to proceed without falsely seeing the warning. If the upgrade process is paused for an extended time period, you can mute the health warning:

ceph health mute DAEMON_OLD_VERSION --sticky

After the upgrade has finished, unmute the health warning:

ceph health unmute DAEMON_OLD_VERSION

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all hosts in the storage cluster.
  • A valid customer subscription.
  • Root-level access to the Ansible administration node.
  • The latest versions of Ansible and ceph-ansible available with Red Hat Ceph Storage 5.
  • The ansible user account for use with the Ansible application.
  • The nodes of the storage cluster is upgraded to Red Hat Enterprise Linux 8.4 or above.
Important

The Ansible inventory file must be present in the ceph-ansible directory.

Procedure

  1. Enable the Ceph and Ansible repositories on the Ansible administration node:

    Syntax

    subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms

  2. On the Ansible administration node, ensure that the latest versions of the ansible and ceph-ansible packages are installed.

    Syntax

    dnf update ansible ceph-ansible

  3. Navigate to the /usr/share/ceph-ansible/ directory:

    Example

    [root@admin ~]# cd /usr/share/ceph-ansible

  4. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, make copies of the group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/osds.yml, and group_vars/clients.yml respectively.

    Example

    [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
    [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml
    [root@admin ceph-ansible]# cp group_vars/rgws.yml.sample group_vars/rgws.yml
    [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

  5. If upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, edit the group_vars/all.yml file to add Red Hat Ceph Storage 5 details.
  6. Once you have done the above two steps, copy the settings from the old yaml files to the new yaml files. Do not change the values of ceph_rhcs_version, ceph_docker_image, and grafana_container_image as the values for these configuration parameters are for Red Hat Ceph Storage 5. This ensures that all the settings related to your cluster are present in the current yaml file.

    Example

    fetch_directory: ~/ceph-ansible-keys
    monitor_interface: eth0
    public_network: 192.168.0.0/24
    ceph_docker_registry_auth: true
    ceph_docker_registry_username: _SERVICE_ACCOUNT_USER_NAME_
    ceph_docker_registry_password: _TOKEN_
    dashboard_admin_user:
    dashboard_admin_password:
    grafana_admin_user:
    grafana_admin_password:
    radosgw_interface: eth0
    ceph_docker_image: "rhceph/rhceph-5-rhel8"
    ceph_docker_image_tag: "latest"
    ceph_docker_registry: "registry.redhat.io"
    node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
    grafana_container_image: registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8:5
    prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
    alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

    Note

    Ensure the Red Hat Ceph Storage 5 container images are set to the default values.

  7. Edit the group_vars/osds.yml file. Add and set the following options:

    Syntax

    nb_retry_wait_osd_up: 50
    delay_wait_osd_up: 30

  8. Open the group_vars/all.yml file and verify the following values are present from the old all.yml file.

    1. The fetch_directory option is set with the same value from the old all.yml file:

      Syntax

      fetch_directory: FULL_DIRECTORY_PATH

      Replace FULL_DIRECTORY_PATH with a writable location, such as the Ansible user’s home directory.

    2. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface option:

      radosgw_interface: INTERFACE

      Replace INTERFACE with the interface to which the Ceph Object Gateway nodes listen.

    3. If your current setup has SSL certificates configured, edit the following:

      Syntax

      radosgw_frontend_ssl_certificate: /etc/pki/ca-trust/extracted/CERTIFICATE_NAME
      radosgw_frontend_port: 443

    4. Uncomment the upgrade_ceph_packages option and set it to True:

      Syntax

      upgrade_ceph_packages: True

    5. If the storage cluster has more than one rgw instance per node, then uncomment the radosgw_num_instances setting and set it to the number of instances per node in the cluster:

      Syntax

      radosgw_num_instances : NUMBER-OF-INSTANCES-PER-NODE

      Example

      radosgw_num_instances : 2

    6. If the storage cluster has RGW multisite defined, check the multisite settings in all.yml to make sure that they contain the same values as they did in the old all.yml file.
  9. Log in as ansible-user on the Ansible administration node.
  10. Execute the rolling_update.yml playbook to convert the storage cluster from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5:

    Syntax

    ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i INVENTORY-FILE

    The -vvvv option collects verbose logs of the upgrade process.

    Example

    [ansible@admin ceph-ansible]$ ansible-playbook -vvvv infrastructure-playbooks/rolling_update.yml -i hosts

    Important

    Using the --limit Ansible option with the rolling_update.yml playbook is not supported.

  11. Review the Ansible playbook log output to verify the status of the upgrade.

Verification

  1. List all running containers:

    Example

    [root@mon ~]# podman ps

  2. Check the health status of the cluster. Replace MONITOR-ID with the name of the Ceph Monitor container found in the previous step:

    Syntax

    podman exec ceph-mon-MONITOR-ID ceph -s

    Example

    [root@mon ~]# podman exec ceph-mon-mon01 ceph -s

  3. Verify the Ceph cluster daemon versions to confirm the upgrade of all daemons. Replace MONITOR-ID with the name of the Ceph Monitor container found in the previous step:

    Syntax

    podman exec ceph-mon-MONITOR-ID ceph --cluster ceph versions

    Example

    [root@mon ~]# podman exec ceph-mon-mon01 ceph --cluster ceph versions

4.10. Converting the storage cluster to using cephadm

After you have upgraded the storage cluster to Red Hat Ceph Storage 5, run the cephadm-adopt playbook to convert the storage cluster daemons to run cephadm.

The cephadm-adopt playbook adopts the Ceph services, installs all cephadm dependencies, enables the cephadm Orchestrator backend, generates and configures the ssh key on all hosts, and adds the hosts to the Orchestrator configuration.

Note

After you run the cephadm-adopt playbook, remove the ceph-ansible package. The cluster daemons no longer work with ceph-ansible. You must use cephadm to manage the cluster daemons.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all nodes in the storage cluster.

Procedure

  1. Log in to the ceph-ansible node and change directory to /usr/share/ceph-ansible.
  2. Run the cephadm-adopt playbook:

    Syntax

    ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i INVENTORY-FILE

    Example

    [ansible@admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/cephadm-adopt.yml -i hosts

  3. Run the following command to enable applications to run on the NFS-Ganesha pool. POOL-NAME is nfs-ganesha, and APPLICATION-NAME is the name of the application you want to enable, such as cephfs, rbd, or rgw.

    Syntax

    ceph osd pool application enable POOL-NAME APPLICATION_NAME

    Example

    [root@host01 ~]# ceph osd pool application enable nfs-ganesha rgw

    Important

    The cephadm-adopt playbook does not bring up rbd-mirroring after migrating the storage cluster from RHCS 4 to RHCS 5.

    To work around this issue, add the peers manually:

    Syntax

    rbd mirror pool peer add POOL_NAME CLIENT_NAME@CLUSTER_NAME

    Example

    [ceph: root@host01 /]# rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b

Additional Resources

4.11. Running the cephadm-ansible playbook on an upgraded storage cluster

Before adding new nodes, new clients, or other new services to your upgraded storage cluster, run the cephadm-ansible.yml playbook. cephadm-ansible.yml is automatically copied to /usr/share/cephadm-ansible during the upgrade process.

Prerequisites

  • Root-level access to all nodes.
  • A valid Red Hat subscription with the appropriate entitlements.
  • An active Red Hat Network (RHN) or service account to access the Red Hat Registry.

Procedure

  1. Install cephadm-ansible:

    Syntax

    dnf install cephadm-ansible

Additional Resources