Chapter 4. Bug fixes
This section describes bugs with significant impact on users that were fixed in this release of Red Hat Ceph Storage. In addition, the section includes descriptions of fixed known issues found in previous versions.
4.1. The Ceph Ansible utility
The size of the replication pool can now be modified after the Ceph cluster deployment
Previously, increasing the size of the replication pool failed after the Ceph cluster was deployed using director. This occurred because an issue with the task in charge of customizing the pool size prevented it from executing when the playbook was rerun. With this update, you can now modify pool size after cluster deployment.
Ceph Ansible supports multiple
grafana instances during a Ceph dashboard deployment
Previously, in a multi-node environment,
ceph-ansible was not able to configure multiple
grafana instances as only one node was supported, leaving the remaining nodes unconfigured. With this update,
ceph-ansible supports multiple instances and injects Ceph-specific layouts on all the Ceph Monitor nodes during the deployment of the Ceph Dashboard.
Running the Ansible
purge-cluster.yml playbook no longer fails when the dashboard feature is disabled
Previously, using the
purge-cluster-yml playbook to purge clusters failed when the dashboard feature was disabled with the following error message:
registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.1 msg: '[Errno 2] No such file or directory'
This occurred because the
dashboard_enabled variable was ignored. With this update, the
dashboard_enabled variable is correctly handled, and
purge-cluster.yml runs successfully.
Red Hat Ceph Storage installation on Red Hat OpenStack Platform no longer fails
ceph-ansible utility became unresponsive when attempting to install Red Hat Ceph Storage with the Red Hat OpenStack Platform 16, and it returns an error similar to the following:
'Error: unable to exec into ceph-mon-dcn1-computehci1-2: no container with name or ID ceph-mon-dcn1-computehci1-2 found: no such container'
This occurred because
ceph-ansible reads the value of the fact
container_exec_cmd from the wrong node in handler_osds.yml
With this update,
ceph-ansible reads the value of
container_exec_cmd from the correct node, and the installation proceeds successfully.
Ansible unsets the
norebalance flag after it completes
Previously, Ansible did not unset the
norebalance flag and it had to be unset manually. With this update, the
rolling-update.yml Ansible playbook unsets the
norebalance flag automatically after it completes and a manual unset is not required.
Ansible upgrades a multisite Ceph Object Gateway when the Dashboard is enabled
Previously, when Red Hat Ceph Storage Dashboard is enabled and an attempt to use Ansible to upgrade to a later version of Red Hat Ceph Storage is made, the upgrade to the secondary Ceph Object Gateway site in a multisite setup failed. With this update to Red Hat Ceph Storage, upgrade of the secondary site works as expected.
Ceph Ansible works with Ansible 2.9
ceph-ansible versions 4.0 and above did not work with Ansible version 2.9. This occurred because the
ceph-validate role did not allow
ceph-ansible to be run against Ansible 2.9. With this update,
ceph-ansible works with Ansible 2.9.
Ceph installations with custom software repositories no longer fail
Previously, using custom repositories to install Ceph were not allowed. This occurred because the
redhat_custom_repository.yml file was removed. With this update, the
redhat_custom_repository.yml file is included, and custom repositories can be used to install Red Hat Ceph Storage.
Only Red Hat-signed packages can use custom software repositories to install Ceph. Custom third-party software repositories are not supported.
ceph-ansible purge playbook does not fail if dashboard was not installed
Previously, when the dashboard was not deployed, the purge playbook failed when purging the cluster because it tried to remove dashboard related resources that did not exist. Consequently, the purge playbook stated that the dashboard is deployed, and purge failed. With this update,
ceph-ansible does not purge dashboard related resources if not part of the deployment, and purge completes successfully.
Using a standalone
nfs-ganesha daemon with an external Ceph storage cluster no longer fails to copy the keyring during deployment
Previously, in configurations consisting of a standalone
nfs-ganesha daemon and an external Ceph storage cluster, the Ceph keyring was not copied to /etc/ceph during a Ceph Ansible deployment.
With this update, the Ceph keyring is copied to /etc/ceph/ directory.
Ceph Ansible updates the privileges of the dashboard admin user after initial install
ceph-ansible could only set the privileges of the dashboard user when it was first created. Running the playbooks after changing
dashboard_admin_user_ro: false from its original setting during install would not update the privileges of the user. In Red Hat Ceph Storage 4.1z1
ceph-ansible has been updated to support changing the dashboard user privileges on successive runs of the playbooks.
docker-to-podman.yml playbook now migrates dashboard containers
Previously, running the
docker-to-podman.yml playbook migrated all the daemons from
podman, except for
grafana-server and the dashboard containers. With this release, running
docker-to-podman.yml successfully migrates all of the daemons.
Storage directories from old containers are removed
Previously, storage directories for old containers were not removed. This could cause high disk usage. This could be seen if you installed Red Hat Ceph Storage, purged it, and reinstalled it. In Red Hat Ceph Storage 4.1z1, storage directories for containers that are no longer being used are removed and excessive disk usage does not occur.
Upgrading a containerized cluster from 4.0 to 4.1 on Red Hat Enterprise Linux 8.1 no longer fails
Previously, when upgrading a Red Hat Ceph Storage cluster from 4.0 to 4.1 the upgrade could fail with an error on
set_fact ceph_osd_image_repodigest_before_pulling. Due to an issue with how the container image tag was updated,
ceph-ansible could fail. In Red Hat Ceph Storage 4.1z1
ceph-ansible has been updated so it no longer fails and upgrading works as expected.
Enabling the Ceph Dashboard fails on an existing OpenStack environment
In an existing OpenStack environment, when configuring the Ceph Dashboard’s IP address and port after the Ceph Manager dashboard module was enabled, was causing a conflict with the HAProxy configuration. To avoid this conflict, configure the Ceph Dashboard’s IP address and port before enabling the Ceph Manager dashboard module.
Red Hat Ceph Storage Dashboard fails when deploying a Ceph Object Gateway secondary site
Previously, the Red Hat Ceph Storage Dashboard would fail to deploy the secondary site in a Ceph Object Gateway multi-site deployment, because when Ceph Ansible ran the
radosgw-admin user create command, the command would return an error. With this release, the Ceph Ansible task in the deployment process has been split into two different tasks. Doing this allows the Red Hat Ceph Storage Dashboard to deploy a Ceph Object Gateway secondary site successfully.
The Ceph File System Metadata Server installation fails when running a playbook with the
Some facts were not getting set on the first Ceph Monitor, but those facts were getting set on all respective Ceph Monitor nodes. When running a playbook with the
--limit option, these facts were not set on the Ceph Monitor, if the Ceph Monitor was not part of the batch. This would cause the playbook to fail when these facts where used in a task for the Ceph Monitor. With this release, these facts are set on the Ceph Monitor whether the playbook uses the
--limit option or not.
Adding a new Ceph Ojbect Gateway instance when upgrading fails
radosgw_frontend_port option did not consider more than one Ceph Object Gateway instance, and configured port
8080 to all instances. With this release, the
radosgw_frontend_port option is increased for each Ceph Object Gateway instance, allowing you to use more than one Ceph Object Gateway instance.
shrink-osd.yml playbook fails when using FileStore in a containerized environment
A default value was missing in Ceph Ansible’s
shrink-osd.yml playbook, which was causing a failure when shrinking a FileStore-backed Ceph OSD in a containerized environment. A previously prepared Ceph OSD using
dmcrypt, was leaving the
encrypted key undefined in the corresponding Ceph OSD file. With this release, a default value was added so the Ceph Ansible
shrink-osd.yml playbook can ran on Ceph OSD that have been prepared using
dmcrypt in containerized environments.
Using HTTPS breaks access to Prometheus and the alert manager
dashboard_protocol option to
https was causing the Red Hat Ceph Storage Dashboard to try and access the Prometheus API, which does not support TLS natively. With this release, Prometheus and the alert manager are force to use the HTTP protocol, when setting the
dashboard_protocol option to
The Ceph Ansible
shrink-osd.yml playbook does not clean the Ceph OSD properly
zap action done by the
ceph_volume module does not handle the
osd_fsid parameter. This caused the Ceph OSD to be improperly zapped by leaving logical volumes on the underlying devices. With this release, the
zap action properly handles the
osd_fsid parameter, and the Ceph OSD can be cleaned properly after shrinking.
The Red Hat Ceph Storage rolling update fails when multiple storage clusters exist
Running the Ceph Ansible
rolling_update.yml playbook when multiple storage clusters are configured, would cause the rolling update to fail because a storage cluster name could not be specified. With this release, the
rolling_update.yml playbook uses the
--cluster option to allow for a specific storage cluster name.
hosts field has an invalid value when doing a rolling update
A Red Hat Ceph Storage rolling update fails because the syntax changed in the evaluation of the
hosts value in the Ceph Ansible
rolling_update.yml playbook. With this release, a fix to the code updates the syntax properly when the
hosts field is specified in the playbook.
rolling_update.yml playbook does not retrieve the storage cluster
When running the
rolling_update.yml playbook, and the Ceph Ansible inventory does not have Ceph Monitor nodes defined, for example, in an external scenario, the storage cluster
fsid is not retrieved. This causes the
rolling_update.yml playbook to fail. With this release, the
fsid retrieval is skipped when there are no Ceph Monitors defined in the inventory, allowing the
rolling_update.yml playbook to execute when no Ceph Monitors are present.
4.2. The Cockpit Ceph installer
Cockpit Ceph Installer no longer deploys Civetweb instead of Beast for RADOS Gateway
Previously, the Cockpit Ceph Installer configured RADOS Gateway (RGW) to use the deprecated Civetweb frontend instead of the currently supported Beast front end. With this update to Red Hat Ceph Storage, the Cockpit Ceph Installer deploys the Beast frontend with RGW as expected.
ansible-runner-service.sh script no longer fails due to a missing repository
Previously, the Cockpit Ceph Installer startup script could fail due to a missing repository in
/etc/containers/registries.conf. The missing repository was
registry.redhat.io. In Red Hat Ceph Storage 4.1z1, the
ansible-runner-service.sh script has been updated to explicitly state the registry name so the repository does not have to be included in
Cockpit Ceph Installer no longer fails on physical network devices with bridges
Previously, the Cockpit Ceph Installer failed if physical network devices were used in a Linux software bridge. This was due to a logic error in the code. In Red Hat Ceph Storage 4.1z1, the code has been fixed and you can use Cockpit Ceph Installer to deploy on nodes with bridges on the physical network interfaces.
Cluster installation no longer fails due to
cockpit-ceph-installer not setting admin passwords for dashboard and grafana
cockpit-ceph-installer did not allow you to set the admin passwords for dashboard and Grafana. This caused storage cluster configuration to fail because
ceph-ansible requires the default passwords to be changed.
With this update,
cockpit-ceph-installer allows you to set the admin passwords in Cockpit so the storage cluster configuration can complete successfully.
Cockpit Ceph Installer allows RPM installation type on Red Hat Enterprise Linux 8
Previously, on Red Hat Enterprise Linux 8, the Cockpit Ceph Installer would not allow you to select RPM for Installation type, you could only install containerized. In Red Hat Ceph Storage 4.1z1, you can select RPM to install Ceph on bare-metal.
4.3. Ceph File System
Improved Ceph File System performance as the number of subvolume snapshots increases
Previously, creating more than 400 subvolume snapshots was degrading the Ceph File System performance by slowing down file system operations. With this release, you can configure subvolumes to only support subvolume snapshots at the subvolume root directory, and you can prevent cross-subvolume links and renames. Doing this allows for the creation of higher numbers of subvolume snapshots, and does not degrade the Ceph File System performance.
Big-endian systems failed to decode metadata for Ceph MDS
Previously, decoding the Ceph MDS metadata on big-endian systems would fail. This was caused by Ceph MDS ignoring the endianness when decode structures from RADOS. The Ceph MDS metadata routines were fixed to correct this issue, resulting in Ceph MDS decoding the structure correctly.
4.4. Ceph Manager plugins
Ceph Manager crashes when setting the alerts interval
There was a code bug in the alerts module for Ceph Manager, which was causing the Ceph Manager to crash. With this release, this code bug was fixed, and you can set the alerts interval without the Ceph Manager crashing.
4.5. The Ceph Volume utility
ceph-volume lvm batch command fails with mixed device types
ceph-volume command did not return the expected return code when devices are filtered using the
lvm batch sub-command, and when the Ceph OSD strategy changed. This was causing
ceph-ansible tasks to fail. With this release, the
ceph-volume command returns the correct status code when the Ceph OSD strategy changes, allowing
ceph-ansible to properly check if new Ceph OSDs can be added or not.
ceph-volume command is treating a logical volume as a raw device
ceph-volume command was treating a logical volume as a raw device, which was causing the
add-osds.yml playbook to fail. This was not allowing additional Ceph OSD to be added to the storage cluster. With this release, a code bug was fixed in
ceph-volume so it handles logical volumes properly, and the
add-osds.yml playbook can be used to add Ceph OSDs to the storage cluster.
nfs-ganesha daemon starts normally
Previously, a configuration using
nfs-ganesha with the RADOS backend would not start because the
nfs-ganesha-rados-urls library was missing. This occurred because the
nfs-ganesha library package for the RADOS backend was moved to a dedicated package. With this update, the
nfs-ganesha-rados-urls package is added to the Ceph container image, so the
nfs-ganesha daemon starts successfully.
4.7. Ceph Object Gateway
Ceph Object Gateway properly applies AWS request signing
Previously, the Ceph Object Gateway did not properly apply an AWS request for signing headers, and was generating the following error message:
With this release, the Ceph Object Gateway code was fixed to properly sign headers. This results in the signing request to succeed when requested.
radosgw-admin bucket check command no longer displays incomplete multipart uploads
Previously, running the
radosgw-admin bucket check command displayed incomplete multipart uploads. This could cause confusion for a site admin because the output might have appeared as though the bucket index were damaged. With this update, the command displays only errors and orphaned objects, and the incomplete uploads are filtered out.
Uneven distribution of omap keys with bucket shard objects
In versioned buckets, occasionally delete object operations were unable to fully complete. In this state, the bucket index entries for these objects had their name and instance strings zeroed out. When there was a subsequent reshard, the empty name and instance strings caused the entry to be resharded to shard 0. Entries that did not belong on shard 0 ended up there. This put a disproportionate number of entries on shard 0 and was larger than each other shard. With this release, the name and instance strings are no longer cleared during this portion of the delete operation. If a reshard takes place, the entries that were not fully deleted nonetheless end up on the correct shard and are not forced to shard 0.
Increase in overall throughput of Object Gateway lifecycle processing performance
Previously, Object Gateway lifecycle processing performance was constrained by the lack of parallelism due to the increasing workload of objects or buckets with many buckets or containers in the given environment. With this update, parallelism is in two dimensions, a single object gateway instance can have several lifecycle processing threads, and each thread has multiple work-pool threads executing the lifecycle work. Additionally, this update improved the allocation of
shards to workers, thereby increasing overall throughput.
Bucket tenanting status is interpreted correctly when
rgw_parse_bucket_key is called
Previously, some callers of
radosgw-admin bucket stats, which processed keys in a loop, could incorrectly interpret untenanted buckets as tenanted, if some tenanted buckets were listed. If
rgw_parse_bucket_key was called with a non-empty rgw bucket argument, it would not correctly assign an empty value for bucket::tenant when no tenant was present in the key. In Red Hat Ceph Storage 4.1z1 the bucket tenant member is now cleared if no tenant applies and bucket tenanting status is interpreted correctly.
The Ceph Object Gateway tries to cache and access anonymous user information
Previously, the Ceph Object Gateway tried to fetch anonymous user information for each request that has not been authenticated. This unauthenticated access was causing high load on a single Ceph OSD in the storage cluster. With this release, the Ceph Object Gateway will try not to fetch anonymous user information, resulting in a decrease in latency and load on a single Ceph OSD.
Lifecycle expiration is reported correctly for objects
Previously, incorrect lifecycle expiration could be reported for some objects, due to the presence of a prefix rule. This was caused because the optional prefix restriction in lifecycle expiration rules was ignored when generating expiration headers used in S3 HEAD and GET requests. In Red Hat Ceph Storage 4.1z1, the rule prefix is now part of the expiration header rule matching and lifecycle expiration for objects is reported correctly.
A high number of objects in the
rgw.none bucket stats
The code that calculates stats failed to check, in some cases, whether a bucket index entry referenced an object that already existed. This was causing the bucket stats to be incorrect. With this release, code was added to check for existence, fixing the bucket stats.
A call to an ordered bucket listing gets stuck
A code bug in the bucket ordered list operation could cause, under specific circumstances, this operation to get stuck in a loop and never complete. With this release, this code bug was fixed, and as a result the call to an ordered bucket listing completes as expected.
Life-cycle processing ignores
A variable which is suppose to contain the modification time of objects during parallel life-cycle processing was incorrectly initialized. This caused non-current versions of objects in buckets with a non-current expiration rule to expire before their intended expiration time. With this release, the modificaction time (
mtime) is correctly initialized and propagates to the life-cycle’s processing queue. This results in the non-current expiration to happen after the correct time period.
Parts of some objects were erroneously added to garbage collection
When reading objects using the Ceph Object Gateway, if parts of those objects took more than half of the value, as defined by the
rgw_gc_obj_min_wait option, then their tail object was added to the garbage collection list. Those tail objects in the garbage collection list were deleted, resulting in data loss. With this release, the garbage collection feature meant to delay garbage collection for deleted objects was disabled. As a result, reading objects using the Ceph Object Gateway that are taking a long time are not added to the garbage collection list.
4.8. Multi-site Ceph Object Gateway
The RGW daemon no longer crashes on shutdown
Previously, the RGW process would abort in certain circumstances due to a race condition during radosgw shutdown. One situation this was issue was seen was when deleting objects when using multisite. This was caused by dereferencing unsafe memory. In Red Hat Ceph Storage 4.1z1 unsafe memory is no longer dereferenced and the RGW daemon no longer crashes.
A health warning status is reported when no Ceph Managers or OSDs are in the storage cluster
In previous Red Hat Ceph Storage releases, the storage cluster health status was
HEALTH_OK even though there were no Ceph Managers or OSDs in the storage cluster. With this release, this health status has changed, and will report a health warning if a storage cluster is not set up with Ceph Managers, or if all the Ceph Managers go down. Because Red Hat Ceph Storage heavily relies on the Ceph Manager to deliver key features, it is not advisable to run a Ceph storage cluster without Ceph Managers or OSDs.
ceph config show command displays the correct
ceph config show command only displayed the configuration keys present in the Ceph Monitor’s database, and because the
fsid is a
NO_MON_UPDATE configuration value, the
fsid was not displaying correctly. With this release, the
ceph config show command displays the correct
Small objects and files in RADOS no longer use more space than required
The Ceph Object Gateway and the Ceph file system (CephFS) stores small objects and files as individual objects in RADOS. Previously, objects smaller than BlueStore’s default minimum allocation size (
min_alloc_size) of 16 KB used more space than required. This happened because the earlier default value of BlueStore’s
min_alloc_size was 16 KB for solid state devices (SSDs). Currently, the default value of
min_alloc_size for SSDs is 4 KB. This enables better use of space with no impact on performance.
Slow ops not being logged in cluster logs
Previously, slow ops were not being logged in cluster logs. They were logged in the
mon logs, but lacked the expected level of detail. With this release, slow ops are being logged in cluster logs, at a level of detail that makes the logs useful for debugging.
Backfills are no longer delayed during placement group merging
Previously, in Red Hat Ceph Storage placement group merges could take longer than expected if the acting set for the source and target placement groups did not match before merging. Backfills done when there is a mismatch can appear to stall. In Red Hat Ceph Storage 4.1z1 the code has been updated to only merge placement groups whose acting sets match. This change allows merges to complete without delay.
Ceph Monitors can grow beyond the memory target
Auto-tuning the memory target was only done on the Ceph Monitor leader and not the Ceph Monitors following the leader. This was causing the Ceph Monitor followers to exceed the set memory target, resulting in the Ceph Monitors crashing once its memory was exhausted. With this release, the auto-tuning process applies the memory target for the Ceph Monitor leader and its followers so memory is not exhausted on the system.
Disk space usage does not increase when OSDs are down for a long time
Previously, when an OSD was down for a long time, a large number of osdmaps were stored and not trimmed. This led to excessive disk usage. In Red Hat Ceph Storage 4.1z1, osdmaps are trimmed regardless of whether or not there are down OSDs and disk space is not overused.
Health metrics are correctly reported when
smartctl exits with a non-zero error code
ceph device get-health-metrics command could fail to report metrics if
smartctl exited with a non-zero error code even though running
smartctl directly reported the correct information. In this case a JSON error was reported instead. In Red Hat Ceph Storage 4.1z1, the
ceph device get-health-metrics command reports metrics even if
smartctl exits with a non-zero error code as long as
smartctl itself reports correct information.
Crashing Ceph Monitors caused by a negative time span
Previously, Ceph Monitors could crash when triggered by a monotonic clock going back in time. These crashes caused a negative monotonic time span and triggered an assertion into the Ceph Monitor leading them to crash. The Ceph Monitor code was updated to tolerate this assertion and interprets it as a zero-length interval and not a negative value. As a result, the Ceph Monitor does not crash when this assertion is made.
Improvements to the encoding and decoding of messages on storage clusters
When deploying a Red Hat Ceph Storage cluster containing a heterogeneous architecture, such as x86_64 and s390, could cause system crashes. Also, under certain workloads for CephFS, Ceph Monitors on s390x nodes could crash unexpectedly. With this release, properly decoding
entity_addrvec_t with a marker of
1, properly decoding the
enum types on big-endian systems by using an intermediate integer variable type, and fixed encoding and decoding
float types on big-endian systems. As a result, heterogeneous storage clusters, and Ceph Monitors on s390x nodes no longer crash.
4.10. RADOS Block Devices (RBD)
rbd unmap commands can be issued concurrently and the corresponding RBD block devices are unmapped successfully
Previously, issuing concurrent
rbd unmap commands could result in udev-related event race conditions. The commands would sporadically fail, and the corresponding RBD block devices might remain mapped to their node. With this update, the udev-related event race conditions have been fixed, and the commands no longer fail.