Release notes for Red Hat Ceph Storage 4.3
Chapter 1. Introduction
Red Hat Ceph Storage is a massively scalable, open, software-defined storage platform that combines the most stable version of the Ceph storage system with a Ceph management platform, deployment utilities, and support services.
The Red Hat Ceph Storage documentation is available at https://access.redhat.com/documentation/en/red-hat-ceph-storage/.
Chapter 2. Acknowledgments
Red Hat Ceph Storage version 4.3 contains many contributions from the Red Hat Ceph Storage team. In addition, the Ceph project is seeing amazing growth in the quality and quantity of contributions from individuals and organizations in the Ceph community. We would like to thank all members of the Red Hat Ceph Storage team, all of the individual contributors in the Ceph community, and additionally, but not limited to, the contributions from organizations such as:
- Fujitsu ®
- Yahoo ™
- Ubuntu Kylin
- Mellanox ®
- CERN ™
- Deutsche Telekom
- Mirantis ®
- SanDisk ™
Chapter 3. New features
This section lists all major updates, enhancements, and new features introduced in this release of Red Hat Ceph Storage.
3.1. The Ceph Ansible utility
Users can now purge the dashboard and monitoring stack only
Previously, users could not purge only the Ceph Manager Dashboard and Monitoring stack components such as Alertmanager, Prometheus, Grafana, and node-exporter separately.
With the `purge-dashboard.yml ` playbook, users can remove only the dashboard and the monitoring stack components.
Purging the storage cluster with
osd_auto_discovery: true now purges the cluster and removes the Ceph OSDs
Previously, purging the storage cluster deployed with
osd_auto_discovery: true would not purge the Ceph OSDs. With this release, the purge playbook works as expected and removes the Ceph OSDs when the storage cluster is deployed with
osd_auto_discovery: true scenario.
The Alertmanager configuration is customizable
With this release, you can customize the Alertmanager configuration using the
alertmanager_conf_overrides parameter in the
The Red Hat Ceph Storage Dashboard deployment is supported on a dedicated network
ceph-ansible stated the address that should be used for deploying the dashboard was on the same subnet as the
With this release, you can override the default dedicated subnet for the dashboard by setting the
dashboard_network parameter in the
/group_vars/all.yml file with the CIDR subnet address.
Setting the global NFS options in the configuration file is supported
ceph-ansible would not allow overriding any parameter in the configuration file.
With this release, you can override any parameter in the NFS_CORE_PARAM block section in the
ganesha.conf file by setting the variable
ganesha_core_param_overrides in group_vars/all.yml and update client-related configuration.
ceph-ansible checks for the Ceph Monitor quorum before starting the upgrade
Previously, when the storage cluster was in a HEALTH ERR or HEALTH WARN state due to one of the Ceph monitors being down, the
rolling_upgrade.yml playbook would run. However, the upgrade would fail and the quorum was lost resulting in I/O down or a cluster failure.
With this release, an additional condition occurs where
ceph-ansible checks the Ceph Monitor quorum before starting the upgrade.
The systemd target units for containerized deployments are now supported
Previously, there was no way to stop all Ceph daemons on a node in a containerized deployment.
With this release, systemd target units for containerized deployments are supported and you can stop all the Ceph daemons on a host or specific Ceph daemons similar to bare-metal deployments.
ceph-ansible now checks the relevant release version during an upgrade before executing the playbook
With this release, during a storage cluster upgrade,
ceph-ansible first checks for the relevant release version and the playbook fails with an error message if a wrong Ceph version is provided.
3.2. Ceph Management Dashboard
A new Grafana Dashboard to display graphs for Ceph Object Gateway multi-site setup
With this release, a new Grafana dashboard is now available and displays graphs for Ceph Object Gateway multisite sync performance including two-way replication throughput, polling latency, and unsuccessful replications.
See the Monitoring Ceph object gateway daemons on the dashboard section in the Red Hat Ceph Storage Dashboard Guide for more information.
3.3. Ceph File System
max_concurrent_clones option to configure the number of clone threads
Previously, the number of concurrent clones was not configurable and the default was 4.
With this release, the maximum number of concurrent clones is configurable using the manager configuration option:
ceph config set mgr mgr/volumes/max_concurrent_clones VALUE
Increasing the maximum number of concurrent clones could improve the performance of the storage cluster.
3.4. Ceph Object Gateway
The role name and the role session information is displayed in the ops log for S3 operations
With this release, you get information such as the role name and the role session in the ops log for all the S3 operations that use temporary credentials returned by AssumeRole* operations for debugging and auditing purposes.
3.5. Multi-site Ceph Object Gateway
Data sync logging experienced delays in processing
Previously, data sync logging could be subject to delays in processing large backlogs of log entries.
With this release, data sync includes caching for bucket sync status. The addition of the cache speeds the processing of duplicate datalog entries when a backlog exists.
Chapter 4. Technology previews
This section provides an overview of Technology Preview features introduced or updated in this release of Red Hat Ceph Storage.
Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information on Red Hat Technology Preview features support scope, see https:
4.1. Block Devices (RBD)
Mapping RBD images to NBD images
rbd-nbd utility maps RADOS Block Device (RBD) images to Network Block Devices (NBD) and enables Ceph clients to access volumes and images in Kubernetes environments. To use
rbd-nbd, install the
rbd-nbd package. For details, see the
rbd-nbd(7) manual page.
4.2. Object Gateway
Object Gateway archive site
With this release an archive site is supported as a Technology Preview. The archive site allows you to have a history of versions of S3 objects that can only be eliminated through the gateways associated with the archive zone. Including an archive zone in a multizone configuration allows you to have the flexibility of an S3 object history in only one zone while saving the space that the replicas of the versions S3 objects would consume in the rest of the zones.
Chapter 5. Deprecated functionality
This section provides an overview of functionality that has been deprecated in all minor releases up to this release of Red Hat Ceph Storage.
Ubuntu is no longer supported
Installing a Red Hat Ceph Storage 4 cluster on Ubuntu is no longer supported. Use Red Hat Enterprise Linux as the underlying operating system.
Configuring iSCSI gateway using
ceph-ansible is no longer supported
Configuring the Ceph iSCSI gateway by using the
ceph-ansible utility is no longer supported. Use
ceph-ansible to install the gateway and then use the
gwcli utility of the to configure the Ceph iSCSI gateway. For details, see the The Ceph iSCSI Gateway chapter in the Red Hat Ceph Storage Block Device Guide.
ceph-disk is deprecated
ceph-disk utility is no longer supported. The
ceph-volume utility is used instead. For details, see the Why does
ceph-volume replace `ceph-disk` section in the Administration Guide for Red Hat Ceph Storage 4.
FileStore is no longer supported in production
The FileStore OSD back end is now deprecated because the new BlueStore back end is now fully supported in production. For details, see the How to migrate the object store from FileStore to BlueStore section in the Red Hat Ceph Storage Installation Guide.
Ceph configuration file is now deprecated
The Ceph configuration file (
ceph.conf) is now deprecated in favor of new centralized configuration stored in Ceph Monitors. For details, see the The Ceph configuration database section in the Red Hat Ceph Storage Configuration Guide.
Chapter 6. Bug fixes
This section describes bugs with significant impact on users that were fixed in this release of Red Hat Ceph Storage. In addition, the section includes descriptions of fixed known issues found in previous versions.
6.1. The Ceph Ansible utility
Alertmanager does not log errors when self-signed or untrusted certificates are used
Previously, when using untrusted CA certificates, Alertmanager generated many errors in the logs.
With this release, the
ceph-ansible can set the
insecure_skip_verify parameter to
true in the
alertmanager.yml file by setting
alertmanager_dashboard_api_no_ssl_verify: true in the
group_vars/all.yml file when using self-signed or untrusted certificates and the Alertmanager does not log those errors anymore and works as expected.
Use a fully-qualified domain name (FQDN) when HTTPS is enabled in a multi-site configuration
Previously, in a multi-site Ceph configuration,
ceph-ansible would not differentiate between HTTP and HTTPS and set the zone endpoints with the IP address instead of the host name when HTTPS was enabled.
With this release, ceph-ansible uses the fully-qualified domain name (FQDN) instead of the IP address when HTTPS is enabled and the zone endpoints are set with the FQDN and match the TLS certificate CN.
--pid-limits parameter as
-1 for podman and
0 for docker in the systemd file to start the container
Previously, the number of processes allowed to run in containers, 2048 for podman and 4096 for docker, were not sufficient to start some containers which needed to start more processes than these limits.
With this release, you can remove the limit of maximum processes that can be started by adding the
--pid-limits parameter as
-1 for podman and as
0 for docker in the systemd unit files. As a result, the containers start even if you customize the internal processes which might need to run more processes than the default limits.
ceph-ansible pulls the monitoring container images in a dedicated task behind the proxy
ceph-ansible would not pull the monitoring container images such as Alertmanager, Prometheus, node-exporter, and Grafana in a dedicated task and would pull images when the systemd service was started.
With this release,
ceph-ansible supports pulling monitoring container images behind a proxy.
ceph-ansible playbook creates the radosgw system user and works as expected
ceph-ansible playbook failed to create the radosgw system user and failed to deploy the dashboard when
rgw_instances was set at the
group_vars level in a multi-site deployment. This variable is not set on Ceph Monitor nodes and given that this where the tasks are delegated, it failed.
With this release,
ceph-ansible checks all the Ceph Object Gateway instances that are defined and sets a boolean fact to check if at least one instance has the
rgw_zonemaster set to ‘True'. The radosgw system user is created and the playbook works as expected.
The Ansible playbook does not fail when used with
dashboard_server_addr parameter was unset when the Ansible playbook was run with the
--limit option and the playbook would fail if the play target did not match the Ceph Manager hosts in a non-collocated scenario.
With this release, you have to set the
dashboard_server_addr parameter on the Ceph Manager nodes and the playbook works as expected.
6.2. Ceph Management Dashboard
The “Client Connection” panel is replaced with “MGRs” on the Grafana dashboard
Previously, the “Client Connection” panel displayed the Ceph File System information and was not meaningful.
With this release, "MGRs" replaces the "Client Connection" panel and displays the count of the active and standby Ceph Managers.
The Red Hat Ceph Storage Dashboard displays the values for disk IOPS
Previously, the Red Hat Ceph Storage Dashboard would not display the Ceph OSD disk performance in the Hosts tab.
With this release, the Red Hat Ceph Storage Dashboard displays the expected information about the Ceph OSDs, host details, and the Grafana graphs.
6.3. The Ceph Volume utility
add-osd.yml playbook does not fail anymore while creating new OSDs
add-osd.yml playbook would fail when new OSDs were added using
ceph-ansible. This was due to the
ceph-volume lvm batch limitation which does not allow addition of new OSDs in a non-interactive mode.
With this release, the
--report options are not passed to the command-line interface and the
add-osd.yml playbook works as expected when creating new OSDs.
6.4. Ceph Object Gateway
rgw_bucket_quota_soft_threshold parameter is disabled
Previously, the Ceph Object Gateway fetched utilization information from the bucket index if the cached utilization reached
rgw_bucket_quota_soft_threshold causing high operations on the bucket index and slower requests.
This release removes the
rgw_bucket_quota_soft_threshold parameter and uses the cached stats resulting in better performance even if the quota limit is almost reached.
radosgw-admin datalog trim command does not crash while trimming a marker
radosgw-admin datalog trim command would crash when trimming a marker in the current generation from
radosgw-admin due to a logic error.
This release fixes a logic error and log trimming occurs without the
radosgw-admin datalog trim command crashing.
6.5. Ceph Manager plugins
The cluster health changes are no longer committed to persistent storage
Previously, rapid changes to the health of the storage cluster caused excessive logging to the
With this release, the
health_history is not logged to the
ceph.audit.log and cluster health changes are no longer committed to persistent storage.
Chapter 7. Known issues
This section documents known issues found in this release of Red Hat Ceph Storage.
7.1. Ceph Management Dashboard
Disk AVG utilization panel shows N/A on the Red Hat Ceph Storage Dashboard
The Red Hat Ceph Storage Dashboard displays a value of N/A on the Overall host performance AVG disk utilization panel thereby showing incorrect Grafana queries.
7.2. Ceph Object Gateway
Lifecycle processing stuck in “PROCESSING” state for a given bucket
If a Ceph Object Gateway server is unexpectedly restarted when the lifecycle processing is in progress for a given bucket, that bucket does not resume processing lifecycle work for at least two scheduling cycles and is stuck in “PROCESSING” state. This is an expected behavior as it is intended to avoid multiple Ceph Object gateway instances or threads from processing the same bucket simultaneously, especially when the debugging is in progress in production. For future releases, the lifecycle processing shall restart on the following day, when debugging is not enabled.
Chapter 8. Sources
The updated Red Hat Ceph Storage source code packages are available at the following location:
- For Red Hat Enterprise Linux 7: http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHCEPH/SRPMS/
- For Red Hat Enterprise Linux 8: http://ftp.redhat.com/redhat/linux/enterprise/8Base/en/RHCEPH/SRPMS/