Chapter 5. Upgrading Ceph Storage Cluster

There are two main upgrading paths:

  • from Red Hat Ceph Storage 1.3 to 2 (Important)
  • between minor versions of Red Hat Ceph Storage 2 or between asynchronous updates (Important)

5.1. Upgrading from Red Hat Ceph Storage 1.3 to 2

Important

Please contact Red Hat support prior to upgrading, if you have a large Ceph Object Gateway storage cluster with millions of objects present in buckets.

For more details refer to the Red Hat Ceph Storage 2.5 Release Notes, under the Slow OSD startup after upgrading to Red Hat Ceph Storage 2.5 heading.

You can upgrade the Ceph Storage Cluster in a rolling fashion and while the cluster is running. Upgrade each node in the cluster sequentially, only proceeding to the next node after the previous node is done.

Red Hat recommends upgrading the Ceph components in the following order:

  • Monitor nodes
  • OSD nodes
  • Ceph Object Gateway nodes
  • All other Ceph client nodes

Two methods are available to upgrade a Red Hat Ceph Storage 1.3.2 to 2.0:

  • Using Red Hat’s Content Delivery Network (CDN)
  • Using a Red Hat provided ISO image file

After upgrading the storage cluster you might have a health warning regarding the CRUSH map using legacy tunables. See the Red Hat Ceph Storage Strategies Guide for more information.

Example

$ ceph -s
    cluster 848135d7-cdb9-4084-8df2-fb5e41ae60bd
     health HEALTH_WARN
            crush map has legacy tunables (require bobtail, min is firefly)
     monmap e1: 1 mons at {ceph1=192.168.0.121:6789/0}
            election epoch 2, quorum 0 ceph1
     osdmap e83: 2 osds: 2 up, 2 in
      pgmap v1864: 64 pgs, 1 pools, 38192 kB data, 17 objects
            10376 MB used, 10083 MB / 20460 MB avail
                  64 active+clean

Important

Red Hat recommends all Ceph clients to be running the same version as the Ceph storage cluster.

5.1.1. Upgrading a Ceph Monitor Node

Red Hat recommends a minimum of three Monitors for a production storage cluster. There must be an odd number of Monitors. While you are upgrading one Monitor, the storage cluster will still have quorum.

For Red Hat Ceph Storage 1.3.2 Monitor nodes running on Red Hat Enterprise Linux 7, perform the following steps on each Monitor node in the storage cluster. Sequentially upgrading one Monitor node at a time.

  1. As root, disable any Red Hat Ceph Storage 1.3.x repositories: 

    # subscription-manager repos --disable=rhel-7-server-rhceph-1.3-mon-rpms --disable=rhel-7-server-rhceph-1.3-installer-rpms --disable=rhel-7-server-rhceph-1.3-calamari-rpms
    Note

    If an ISO-based installation was performed for Red Hat Ceph Storage 1.3.x, then skip this first step.

  2. Enable the Red Hat Ceph Storage 2 Monitor repository. For ISO-based installations, see the ISO installation section.

  3. As root, stop the Monitor process:

    Syntax

    # service ceph stop <daemon_type>.<monitor_host_name>

    Example

    # service ceph stop mon.node1

  4. As root, update the ceph-mon package: 

    # yum update ceph-mon

  5. As root, update the owner and group permissions:

    Syntax

    # chown -R <owner>:<group> <path_to_directory>

    Example

    # chown -R ceph:ceph /var/lib/ceph/mon
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown ceph:ceph /etc/ceph/ceph.client.admin.keyring
# chown ceph:ceph /etc/ceph/ceph.conf
# chown ceph:ceph /etc/ceph/rbdmap

    Note

    If the Ceph Monitor node is co-located with an OpenStack Controller node, then the Glance and Cinder keyring files must be owned by glance and cinder respectively. For example: 

    # ls -l /etc/ceph/
...
-rw-------.  1 glance glance      64 <date> ceph.client.glance.keyring
-rw-------.  1 cinder cinder      64 <date> ceph.client.cinder.keyring
...

  6. If SELinux is set to enforcing or permissive mode, then set a relabelling of the SELinux context on files for the next reboot: 

    # touch /.autorelabel
    Warning

    Relabeling will take a long time to complete, because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabelled, add the directory to the /etc/selinux/fixfiles_exclude_dirs file before rebooting.

Note

In environments with large number of objects per placement group (PG), the directory enumeration speed will decrease, causing a negative impact to performance. This is caused by the addition of xattr queries which verifies the SELinux context. Setting the context at mount time removes the xattr queries for context and helps overall disk performance, especially on slower disks.

Add the following line to the [osd] section in the /etc/ceph/ceph.conf file: 

osd_mount_options_xfs=rw,noatime,inode64,context="system_u:object_r:ceph_var_lib_t:s0"

  1. As root, replay device events from the kernel: 

    # udevadm trigger

  2. As root, enable the ceph-mon process: 

    # systemctl enable ceph-mon.target
# systemctl enable ceph-mon@<monitor_host_name>

  3. As root, reboot the Monitor node: 

    # shutdown -r now

  4. Once the Monitor node is up, check the health of the Ceph storage cluster before moving to the next Monitor node: 

    # ceph -s

To add more Red Hat Ceph Storage Monitors to the storage cluster, see the Red Hat Ceph Storage Administration Guide

5.1.2. Upgrading a Ceph OSD Node

Red Hat recommends having a minimum of three OSD nodes in the Ceph storage cluster. For Red Hat Ceph Storage 1.3.2 OSD nodes running on Red Hat Enterprise Linux 7, perform the following steps on each OSD node in the storage cluster. Sequentially upgrading one OSD node at a time.

During the upgrade of an OSD node, some placement groups will become degraded, because the OSD might be down or restarting. You will need to tell the storage cluster not to mark an OSD out, because you do not want to trigger a recovery. The default behavior is to mark an OSD out of the CRUSH map after five minutes.

On a Monitor node, set noout and norebalance flags for the OSDs: 

# ceph osd set noout
# ceph osd set norebalance

Perform the following steps on each OSD node in the storage cluster. Sequentially upgrading one OSD node at a time. If an ISO-based installation was performed for Red Hat Ceph Storage 1.3, then skip this first step.

  1. As root, disable the Red Hat Ceph Storage 1.3 repositories: 

    # subscription-manager repos --disable=rhel-7-server-rhceph-1.3-osd-rpms --disable=rhel-7-server-rhceph-1.3-installer-rpms --disable=rhel-7-server-rhceph-1.3-calamari-rpms
  2. Enable the Red Hat Ceph Storage 2 OSD repository. For ISO-based installations, see the ISO installation section.

  3. As root, stop any running OSD process:

    Syntax

    # service ceph stop <daemon_type>.<osd_id>

    Example

    # service ceph stop osd.0

  4. As root, update the ceph-osd package: 

    # yum update ceph-osd

  5. As root, update the owner and group permissions on the newly created directory and files:

    Syntax

    # chown -R <owner>:<group> <path_to_directory>

    Example

    # chown -R ceph:ceph /var/lib/ceph/osd
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown -R ceph:ceph /etc/ceph

    Note

    Running the following find command might speed up the process of changing ownership by running the chown command in parallel on a Ceph storage cluster with a large number of disks: 

    # find /var/lib/ceph/osd -maxdepth 1 -mindepth 1 -print | xargs -P12 -n1 chown -R ceph:ceph

  6. If SELinux is set to enforcing or permissive mode, then set a relabelling of the SELinux context on files for the next reboot: 

    # touch /.autorelabel
    Warning

    Relabeling will take a long time to complete, because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabelled, add the directory to the /etc/selinux/fixfiles_exclude_dirs file before rebooting.

Note

In environments with large number of objects per placement group (PG), the directory enumeration speed will decrease, causing a negative impact to performance. This is caused by the addition of xattr queries which verifies the SELinux context. Setting the context at mount time removes the xattr queries for context and helps overall disk performance, especially on slower disks.

Add the following line to the [osd] section in the /etc/ceph/ceph.conf file: 

osd_mount_options_xfs=rw,noatime,inode64,context="system_u:object_r:ceph_var_lib_t:s0"

  1. As root, replay device events from the kernel: 

    # udevadm trigger

  2. As root, enable the ceph-osd process: 

    # systemctl enable ceph-osd.target
# systemctl enable ceph-osd@<osd_id>

  3. As root, reboot the OSD node: 

    # shutdown -r now

  4. Move to the next OSD node.

    Note

    While the noout and norebalance flags are set, the storage cluster will have the HEALTH_WARN status: 

    $ ceph health
HEALTH_WARN noout,norebalance flag(s) set

Once you are done upgrading the Ceph storage cluster, unset the previously set OSD flags and verify the storage cluster status.

On a Monitor node, and after all OSD nodes have been upgraded, unset the noout and norebalance flags: 

# ceph osd unset noout
# ceph osd unset norebalance

In addition, set the require_jewel_osds flag. This flag ensures that no more OSDs with Red Hat Ceph Storage 1.3 can be added to the storage cluster. If you do not set this flag, the storage status will be HEALTH_WARN. 

# ceph osd set require_jewel_osds

To expand the storage capacity by adding new OSDs to the storage cluster, see the Red Hat Ceph Storage Administration Guide for more details.

5.1.3. Upgrading the Ceph Object Gateway Nodes

This section describes steps to upgrade a Ceph Object Gateway node to a later version.

Before You Start
  • Red Hat recommends putting a Ceph Object Gateway behind a load balancer, such as HAProxy. If you use a load balancer, remove the Ceph Object Gateway from the load balancer once no requests are being served.

  • If you use a custom name for the region pool, specified in the rgw_region_root_pool parameter, add the rgw_zonegroup_root_pool parameter to the [global] section of the Ceph configuration file. Set the value of rgw_zonegroup_root_pool to be the same as rgw_region_root_pool, for example: 

    [global]
rgw_zonegroup_root_pool = .us.rgw.root
Procedure: Upgrading the Ceph Object Gateway Node

  1. If you used online repositories to install Red Hat Ceph Storage, disable the 1.3 repositories. 

    # subscription-manager repos --disable=rhel-7-server-rhceph-1.3-tools-rpms --disable=rhel-7-server-rhceph-1.3-installer-rpms --disable=rhel-7-server-rhceph-1.3-calamari-rpms
  2. Enable the Red Hat Ceph Storage 2 Tools repository. For ISO-based installations, see the ISO Installation section.

  3. Stop the Ceph Object Gateway process (ceph-radosgw): 

    # service ceph-radosgw stop

  4. Update the ceph-radosgw package: 

    # yum update ceph-radosgw

  5. Change the owner and group permissions on the newly created /var/lib/ceph/radosgw/ and /var/log/ceph/ directories and their content to ceph. 

    # chown -R ceph:ceph /var/lib/ceph/radosgw
# chown -R ceph:ceph /var/log/ceph

  6. If SELinux is set to run in enforcing or permissive mode, instruct it to relabel SELinux context on the next boot. 

    # touch /.autorelabel
    Important

    Relabeling takes a long time to complete, because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabeled, add them to the /etc/selinux/fixfiles_exclude_dirs file before rebooting.

Note

In environments with large number of objects per placement group (PG), the directory enumeration speed will decrease, causing a negative impact to performance. This is caused by the addition of xattr queries which verifies the SELinux context. Setting the context at mount time removes the xattr queries for context and helps overall disk performance, especially on slower disks.

Add the following line to the [osd] section in the /etc/ceph/ceph.conf file: 

osd_mount_options_xfs=rw,noatime,inode64,context="system_u:object_r:ceph_var_lib_t:s0"

  1. Enable the ceph-radosgw process. 

    # systemctl enable ceph-radosgw.target
# systemctl enable ceph-radosgw@rgw.<hostname>

    Replace <hostname> with the name of the Ceph Object Gateway host, for example gateway-node. 

    # systemctl enable ceph-radosgw.target
# systemctl enable ceph-radosgw@rgw.gateway-node

  2. Reboot the Ceph Object Gateway node. 

    # shutdown -r now
  3. If you use a load balancer, add the Ceph Object Gateway node back to the load balancer.
  4. Repeat these steps on a next Ceph Object Gateway node.
See Also

5.1.4. Upgrading a Ceph Client Node

Ceph clients can be the RADOS Gateway, RADOS block devices, the Ceph command-line interface (CLI), Nova compute nodes, qemu-kvm, or any custom application using the Ceph client-side libraries. Red Hat recommends all Ceph clients to be running the same version as the Ceph storage cluster.

Important

Red Hat recommends stopping all IO running against a Ceph client node while the packages are being upgraded. Not stopping all IO might cause unexpected errors to occur.

  1. As root, disable any Red Hat Ceph Storage 1.3 repositories: 

    # subscription-manager repos --disable=rhel-7-server-rhceph-1.3-tools-rpms --disable=rhel-7-server-rhceph-1.3-installer-rpms --disable=rhel-7-server-rhceph-1.3-calamari-rpms
    Note

    If an ISO-based installation was performed for Red Hat Ceph Storage 1.3.x clients, then skip this first step.

  2. On the client node, enable the Tools repository.

  3. On the client node, update the ceph-common package: 

    # yum update ceph-common

Any application depending on the Ceph client-side libraries will have to be restarted after upgrading the Ceph client package.

Note

For Nova compute nodes with running qemu-kvm instances or if using a dedicated qemu-kvm client, then stopping and starting the qemu-kvm instance processes is required. A simple restart will not work here.

5.2. Upgrading Between Minor Versions and Applying Asynchronous Updates

Important

Please contact Red Hat support prior to upgrading, if you have a large Ceph Object Gateway storage cluster with millions of objects present in buckets.

For more details refer to the Red Hat Ceph Storage 2.5 Release Notes, under the Slow OSD startup after upgrading to Red Hat Ceph Storage 2.5 heading.

Use the Ansible rolling_update.yml playbook located in the infrastructure-playbooks directory from the administration node to upgrade between two minor versions of Red Hat Ceph Storage 2 or to apply asynchronous updates.

Currently, this is the only supported way to upgrade to a minor version. If you use a cluster that was not deployed by using Ansible, see Section 3.1.7, “Taking over an Existing Cluster” for details on configuring Ansible to use a cluster that was deployed without it.

Ansible upgrades the Ceph nodes in the following order:

  • Monitor nodes
  • OSD nodes
  • MDS nodes
  • Ceph Object Gateway nodes
  • All other Ceph client nodes
Note

Upgrading encrypted OSD nodes is the same as upgrading OSD nodes that are not encrypted.

Before you Start

  • On the Ansible Administration node, enable the Red Hat Ceph Storage 2 Tools repository: 

    # subscription-manager repos --enable=rhel-7-server-rhceph-2-tools-rpms

  • On the Ansible Administration node, ensure the latest version of ceph-ansible is installed: 

    # yum update ceph-ansible

  • In the rolling_update.yml playbook, change the health_osd_check_retries and health_osd_check_delay values to 40 and 30 respectively. For each OSD node, Ansible will wait up to 20 minutes. Also, Ansible will check the cluster health every 30 seconds, waiting before continuing the upgrade process. Set the following values: 

    health_osd_check_retries: 40
health_osd_check_delay: 30
  • If the Ceph nodes are not connected to the Red Hat Content Delivery Network (CDN) and you used an ISO image to install Red Hat Ceph Storage, update the local repository with the latest version of Red Hat Ceph Storage. See Section 2.3, “Enabling the Red Hat Ceph Storage Repositories” for details.
  • If you upgrade from Red Hat Ceph Storage 2.1 to 2.2, review the Section 5.2.1, “Changes Between Ansible 2.1 and 2.2” section first. Ansible 2.2 uses slightly different file names and setting.

Procedure: Updating the Ceph Storage Cluster by using Ansible

  1. On the Ansible administration node, edit the /etc/ansible/hosts file with custom osd_scenarios if your cluster has any.

  2. On the Ansible administration node, navigate to the /usr/share/ceph-ansible/ directory: 

    # cd /usr/share/ceph-ansible

  3. In the group_vars/all.yml file, uncomment the upgrade_ceph_packages option and set it to True: 

    upgrade_ceph_packages: True
  4. In the group_vars/all.yml file, set generate_fsid to false.
  5. Get the current cluster fsid by executing ceph fsid. Set the retrieved fsid in group_vars/all.yml.

  6. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface parameter to the group_vars/all.yml file. 

    radosgw_interface: <interface>

    Replace:

    • <interface> with the interface that the Ceph Object Gateway nodes listen to

  7. Run the rolling_update.yml playbook: 

    # cp infrastructure-playbooks/rolling_update.yml .
$ ansible-playbook rolling_update.yml

    When upgrading from version 2.4 to 2.5, run the playbook using the following command: 

    $ ansible-playbook rolling_update.yml -e jewel_minor_update=true

  8. From the RBD mirroring daemon node, upgrade rbd-mirror manually: 

    # yum upgrade rbd-mirror

    Restart the daemon: 

    # systemctl restart  ceph-rbd-mirror@<client-id>
Important

The rolling_update.yml playbook includes the serial variable that adjusts the number of nodes to be updated simultaneously. Red Hat strongly recommends to use the default value (1), which ensures that hosts will be upgraded one by one.

5.2.1. Changes Between Ansible 2.1 and 2.2

Red Hat Ceph Storage 2.2 includes Ansible 2.2 that introduces the following changes:

  • Files in the group_vars directory have the .yml extension. Before updating to 2.2, you must rename them. To do so:

    Navigate to the Ansible directory: 

    # cd usr/share/ceph-ansible

    Change the names of the files in group_vars: 

    # mv groups_vars/all groups_vars/all.yml
# mv groups_vars/mons groups_vars/mons.yml
# mv groups_vars/osds groups_vars/osds.yml
# mv groups_vars/mdss groups_vars/mdss.yml
# mv groups_vars/rgws groups_vars/rgws.yml

  • Ansible 2.2 uses different variable names and handles this change automatically when updating to version 2.2. See Table 5.1, “Differences in Variable Names Between Ansible 2.1 and 2.2” table for details.

    Table 5.1. Differences in Variable Names Between Ansible 2.1 and 2.2

    Ansible 2.1 variable nameAnsible 2.2 variable name

    ceph_stable_rh_storage

    ceph_rhcs

    ceph_stable_rh_storage_version

    ceph_rhcs_version

    ceph_stable_rh_storage_cdn_install

    ceph_rhcs_cdn_install

    ceph_stable_rh_storage_iso_install

    ceph_rhcs_iso_install

    ceph_stable_rh_storage_iso_path

    ceph_rhcs_iso_path

    ceph_stable_rh_storage_mount_path

    ceph_rhcs_mount_path

    ceph_stable_rh_storage_repository_path

    ceph_rhcs_repository_path