RHBA-2023:7780 - Bug Fix Advisory

Topic

An update is now available for Red Hat Ceph Storage 7.0 in the Red Hat
Ecosystem Catalog.

Description

Red Hat Ceph Storage is a scalable, open, software-defined storage platform
that combines the most stable version of the Ceph storage system with a
Ceph management platform, deployment utilities, and support services.

These new packages include numerous enhancements, bug fixes, and known
issues. Space precludes documenting all of these changes in this advisory.
Users are directed to the Red Hat Ceph Storage Release Notes for
information on the most significant of these changes:

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/7.0/html/release_notes/index

Solution

Before applying this update, make sure all previously released errata
relevant to your system have been applied.

For details on how to apply this update, refer to:

https://access.redhat.com/articles/11258

Affected Products

• Red Hat Enterprise Linux for x86_64 9 x86_64
• Red Hat Enterprise Linux for IBM z Systems 9 s390x
• Red Hat Enterprise Linux for Power, little endian 9 ppc64le

Fixes

• BZ - 1740782 - [Workload-DFG] RGW Multisite Sync fairness imbalance
• BZ - 1846154 - hard to tell if encryption is on or off
• BZ - 1898647 - rgw gc list slow to respond
• BZ - 1929346 - [cee/sd][RFE][RGW] Add operation (GET, PUT, DELETE, ...) counters/trackers in RadosGW per bucket and users
• BZ - 1962087 - [Cephadm] Cephadm is not reflecting updated values to ceph status
• BZ - 1994251 - [RFE][GSS] Need ssl between node-exporter, Prometheus and mgr module
• BZ - 2008835 - [GSS][RGW]Arbitrarily-large space leaks generated by re-uploading the same multi-part part multiple times
• BZ - 2028335 - [RFE][cephadm] Allow the configuration of prometheus authentication using cephadm
• BZ - 2032001 - ceph rgw used 80+G memory
• BZ - 2049445 - Same host added multiple times via " ceph orch host add " command
• BZ - 2059839 - [cee/sd][cephadm] cephadm bootstrap is not applying the configuration which passed via the option "--apply-spec"
• BZ - 2068491 - [RGW][RFE]include versioning details of the bucket in radosgw-admin bucket stats command
• BZ - 2091154 - RFE: change Ceph quota sizes from bytes to G or M or T
• BZ - 2095636 - [RFE] Cephadm raw OSD support with "apply OSD" command
• BZ - 2097490 - [RFE] Support HAProxy's PROXY protocol
• BZ - 2106031 - [cee/sd][ceph-mgr]ceph-mgr daemon got lost from ceph status
• BZ - 2107339 - [cephadm] ceph orch host maintenance enter ignores the --force flag
• BZ - 2110933 - [RFE] : ceph versions : mds : remove empty list entries from ceph versions
• BZ - 2111539 - [cee/sd][ceph-mgr] While setting host into maintenance, getting a traceback "raise OrchestratorError(msg, errno=rc)"
• BZ - 2113901 - [cephadm] host label remove not validating labels
• BZ - 2115184 - Seeing "CEPHADM_STRAY_HOST" health warning after running cephadm adopt playbook
• BZ - 2115462 - extend error message when hitting execnet exception on closed I/O
• BZ - 2118726 - [RGW][s3-select][parquet]: query on parquet object without .parquet extension fails with " out of memory " error
• BZ - 2125382 - [cee][cephadm] ceph orch ps does not display version tag for alertmanager/node-exporter/prometheus
• BZ - 2125709 - Unnecessarily long laggy PG state
• BZ - 2126465 - [orch] Stopping crash daemon/service results in the service being marked as error instead of stopped
• BZ - 2128421 - [RGW][archive]: recovering shards in sync status on archive site upon a bucket delete from primary
• BZ - 2129414 - [cee/sd][BlueFS][RHCS 5.x] no BlueFS spillover health warning in RHCS 5.x
• BZ - 2138224 - [Workload DFG][RGW MS Active-Active] Slow sync in both directions
• BZ - 2140535 - [RFE] : osd spec : orchestrator could recognise errors on indentation and provide easier inputs to user
• BZ - 2148242 - [minor issue] : rados cppool : extra tab spaces in error messages
• BZ - 2149453 - [RFE] BLK/Kernel: Improve protection against running one OSD twice
• BZ - 2151171 - [ceph dashboard] provide image usage in the form of progress bar in image section of block similar to pool usage
• BZ - 2153827 - `ceph orch host drain` should not always delete contents of /etc/ceph/
• BZ - 2155380 - [RFE] Set single compression algorithm as a default value in ms_osd_compression_algorithm instead of list of algorithms
• BZ - 2155625 - [RFE] Need support to set unmanaged flag for OSD service other than all-available-devices
• BZ - 2159966 - [RGW]: Granular sync group create command allows invalid names
• BZ - 2160839 - [Workload-DFG] [RGW MultiSite] :multiple objects are not synced from secondary site to primary
• BZ - 2163667 - [RGW][kafka][LC]: notification of 'AbortMultipartUpload' is not seen when lc rule removes incomplete multipart upload and rgw daemon crashes
• BZ - 2166705 - [dashboard] block/rbd : images -> snapshots : disable protect option if layering feature is not enabled on the image
• BZ - 2166708 - [dashboard] block/rbd : images -> edit -> disable checkboxes for layering and deef-flatten on pre-created image
• BZ - 2172795 - PG Autoscaler does not throw warn to increase PG count on pools with autoscale_mode set to warn
• BZ - 2176300 - [RFE] Configure Ingress and NFS to support HAProxy's PROXY protocol
• BZ - 2180415 - [RGW]bucketnotifiaction failed with AccessDenied for same and diffent tenant users using bucket policy
• BZ - 2182421 - Trino / Ceph integration requires changes in S3select engine and RGW
• BZ - 2189412 - [RGW][The value of the content-length header is inconsistent with the body size for website-enabled buckets]
• BZ - 2190355 - Integrate RGW Multi-site with ceph dashboard
• BZ - 2192563 - [RFE] : login-change-password : Confusing green tick for old password field
• BZ - 2214981 - [CEE/sd][RGW] RGWSI_Notify::robust_notify(const DoutPrefixProvider*, RGWSI_RADOS::Obj&, const RGWCacheNotifyInfo&, optional_yield):402 Notify failed on object: (110) Connection timed out
• BZ - 2216442 - [NFS-Ganesha] File locking test from multiple nfs client on same file is failing with ganesha service crash
• BZ - 2217540 - [NFS-Ganesha] Ganesha process getting crashed while writing from client 1 and performing lookup from client 2
• BZ - 2218013 - CPU and latency spikes during periodic snaptrim operations
• BZ - 2219093 - MGR process hung - pybind/cephfs: holds GIL during rmdir
• BZ - 2219427 - rgw: take upstream fixes for race handling zone trace during full sync
• BZ - 2219467 - rgw: take fixes for inconsistent updates of OLH and bucket index objects
• BZ - 2220924 - [Ceph Dashboard]: rhcs 7.0 downstream branding
• BZ - 2222010 - [GSS] RGW HA ingress service activates virtual IP on all instances at the same time
• BZ - 2223249 - NFS server is getting crashed and moving to error state
• BZ - 2224078 - radosgw-admin generated core-dump after fill workload using Warp
• BZ - 2224241 - client: trigger to flush the buffer when making snapshot
• BZ - 2225434 - rgw crashes seen for s3select json query with "where" clause
• BZ - 2226730 - Getting 500 - Internal Server Error pop up during Zone group rename even though operation is success
• BZ - 2226732 - Unable to remove non-master zone from the zone group via dashboard
• BZ - 2227987 - client: wait rename to finish
• BZ - 2227988 - client: do not send metrics until the MDS rank is ready
• BZ - 2227992 - mds: cap revoke and cap update's seqs mismatched
• BZ - 2228000 - client: issue a cap release immediately if no cap exists
• BZ - 2228003 - mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg
• BZ - 2228066 - mds: do not evict clients if OSDs are laggy
• BZ - 2228157 - radosgw-admin crashes when using --placement-id
• BZ - 2228260 - update nfs-ganesha to V5.6 in RHCS 7.0
• BZ - 2228358 - mds: MDLog::_recovery_thread: handle the errors gracefully
• BZ - 2228635 - (mds.1): 3 slow requests are blocked
• BZ - 2228874 - [RGW] revised fix for Notify timeout can induce a race condition as it attempts to resend the cache update
• BZ - 2229566 - [cephadm][upgrade]: mgr crash seen and ceph orch commands fail with "Error ENOENT: Module not found" on upgrade from 6.1(GA) to 7.0(latest)
• BZ - 2230056 - rbd-mirror daemon may skip deleting non-primary image when primary image is deleted [7.0]
• BZ - 2230076 - Disallow enabling snapshot-based mirroring on cloned images [7.0]
• BZ - 2230230 - [s3select][json]: rgw crashes seen for query "select phonenumbers.type from s3object[*].phonenumbers;"
• BZ - 2230233 - [s3select][json]: rgw crashes seen for query "select min() from s3object[*].phonenumbers;"
• BZ - 2230234 - [s3select][json]: rgw crashes seen for query "select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL;"
• BZ - 2231342 - Ceph dashboard allows to set invalid value for quota value for max bytes which leads to wrongful setting of quota
• BZ - 2231452 - RFE: CU can not deploy multiple RGW ingress service with different virtual IPs and the same port
• BZ - 2232295 - [RFE] [Ceph-Dashboard]: Cluster upgrade management
• BZ - 2232297 - [RFE] [Ceph-Dashboard]: CephFS Volume management
• BZ - 2232298 - [RFE] [Ceph-Dashboard]: CephFS Subvolume management
• BZ - 2232300 - [RFE] [Ceph Dashboard]: Improve the bucket details to see encryption locking, versioning
• BZ - 2232492 - [Ceph Dashboard] About modal displaying incorrect release version
• BZ - 2232567 - [Ceph Dashboard] Host Details section under the Expand Cluster review page is displaying blank space
• BZ - 2232877 - One of the ceph mgr module crashed while performing host OS upgrade
• BZ - 2233356 - Rgw overview dashboard in ceph dashboard
• BZ - 2233800 - avoid ballooning client_mount_timeout by 10x (5 -> 50 minutes) [7.0]
• BZ - 2235338 - [GSS][cephfs] CephFS blocked requests with warning 1 MDSs behind on trimming
• BZ - 2235566 - Toplology view in multi-site dashboard displaying blank table regarding realm, zonegroup and zones in chrome browser
• BZ - 2235592 - [RGW-MS]: Multisite sync status under Object Gateway Overview section is stuck in loading state for secondary site
• BZ - 2235625 - Stale entries are displayed on file system dashboard on unsuccessful volume creation operation
• BZ - 2236109 - SignatureDoesNotMatch error on RGW Object Gateway Dashboard after upgrade to 6.1
• BZ - 2236162 - Dashboard cephfs subvolume removal with snapshots fails
• BZ - 2236325 - Ganesha showing high memory usage (100+ GiB) which is also not being released over time
• BZ - 2236462 - [s3select][json]: wrong output seen for query "select _1.authors.name from s3object[*] limit 1"
• BZ - 2236643 - [rgw-ms][compression-encryption][usage][7.0]: Encrypted multipart uploads cause an inconsistency in size obtained via bucket stats
• BZ - 2236691 - Default images for downstream RHCS 7.0 monitoring stack being pulled from quay instead of redhat registry
• BZ - 2236789 - cephadm: support for CA signed keys
• BZ - 2237303 - [rbd-mirror] : snapshot schedules stopped : possibly due to hang in MirrorSnapshotScheduleHandler.shutdown which could be in wait_for_pending() [7.0]
• BZ - 2237304 - [rbd-mirror] : primary- 'demoted ' snapshots piling up after consecutive planned failovers (relocation) [7.0]
• BZ - 2237424 - [ceph-dashboard] In file system add missing subvolume groups listing and actions in subvolumes
• BZ - 2237490 - [7.0][RGW multisite][Archive zone][Duplicate objects in the archive zone]
• BZ - 2237838 - [7.0][rgw-ms]: Download of encrypted+multipart object fails with AccessDenied (403) on the remote site
• BZ - 2237859 - Filesystem tab does not display - Ranks, Pools and standby details of a file system
• BZ - 2237904 - [7.0][rgw-archive]: Archive zone fixes present in rhcs6.1 not present in rhcs-7.0 latest
• BZ - 2238176 - rgw: fix build of rgw-nfs unit tests
• BZ - 2238272 - [Nfs-Ganesha] Nfs mounts are taking longer time than expected when tried on a scale setup
• BZ - 2238332 - mgr/dashboard: Object gateway inventory card incorrect Buckets and user count
• BZ - 2238333 - mgr/dashboard: Object gateway sync status cards keeps loading when multisite is not configured
• BZ - 2238470 - [Ceph Dashboard]: RGW page is showing 500 error
• BZ - 2238472 - [Ceph Dashboard]: Improvements and query fixing on dashboard utilization graphs
• BZ - 2238663 - mds: blocklist clients with "bloated" session metadata
• BZ - 2238672 - [Ceph-Dashboard] Cluster upgrade management - Cluster Status 'OK' hover displays empty text box
• BZ - 2238921 - [rhcs7.0][RGW-MS][Notification]: bucket owner not in event record and received object size 0 for s3:ObjectSynced:Create event
• BZ - 2238976 - [Workload-DFG] [RGW MultiSite] RGW seg faults during sync
• BZ - 2239114 - [RGW-Multisite]: Importing multisite token at site2 throws Internal Server Error
• BZ - 2239173 - [rhcs7.0][rgw][notifications]:with kafka-ack-level=broker, objects deletion with boto3 failed with read timeout on endpoint url and observed multiple delete notifications sent for single object
• BZ - 2239831 - [rgw-ms][archive]: Observing large omaps on the archive zone for a few buckets uploaded with ~1.65M objects which were not adequately dynamically resharded.
• BZ - 2239888 - [CEE/SD][ceph-volume]Even though there is enough space in the DB device, the OSDs are not being created after attaching the device
• BZ - 2239892 - Deploy NVMEoF containers
• BZ - 2239911 - [rgw] policy array empty on rgw swift /info in reef
• BZ - 2239921 - [rgw-ms][archive]: The overall sync status shows behind shards in the archive zone when executing hybrid workload for 7 hours on the primary and secondary site.
• BZ - 2239992 - [cephfs][dashboard] create subvolume group name with "/" shows it already exists
• BZ - 2239993 - [cephfs][dashboard] create subvolume allows invalid start characters
• BZ - 2240003 - [7.0][RGW-Multisite] Sync fairness: Multiple daemons crash and not coming back up during incremental sync
• BZ - 2240145 - [cephfs][dashboard] When rename volume, it shows alerts about subvolume.
• BZ - 2240169 - Gateway fails to load subsystems upon nvmeof service restart
• BZ - 2240179 - create_listener always successful with wrong inputs ends up in OMAP
• BZ - 2240189 - Post log in to the 7.0 dashboard getting 500 internal error popup
• BZ - 2240280 - [CephFS-NFS] CEPH_FS_NONBLOCKING_IO is stuck when compiling the Linux kernel in the NFS mount director
• BZ - 2240582 - Only hours part in time format is displayed in multisite sync status in dashboard
• BZ - 2240588 - NVMeoF GW deployment failed with rados.PermissionDeniedError error
• BZ - 2240766 - [RFE] RBD mirroring (and related processes) needs a connection health metric to its peers
• BZ - 2240832 - [RADOS] "currently delayed" slow ops does not provide details on why op has been delayed
• BZ - 2240972 - missing ceph_daemon label on labeled perfcounters in ceph-exporter
• BZ - 2241040 - SPDK reactor (pool_group_0) becomes a bottleneck under I/O load
• BZ - 2241126 - HAproxy version is shown as <unknown> in the ceph orch ps command output
• BZ - 2241145 - Changing permission of a directory at nfs mount with rgw backend causes unavailability of a directory for sometime
• BZ - 2241146 - [7.0][rgw-archive]: On archive zone, bucket versioning shows disabled on 7.0 even though it is enabled
• BZ - 2241186 - [RGW-NFS]: Getting Input/output error while copying files to the directory in NFS mount with RGW backend
• BZ - 2241309 - mgr/dashboard: empty grafana panels for performance of daemons
• BZ - 2241325 - [bucket-notification/versioning]: VersionId not present for CurrentVersion ObjectLifecycle:Expiration events
• BZ - 2241346 - Unable to discover subsystems from Initiator with ceph-nvmeof version 0.0.4-1
• BZ - 2241558 - [cephadm] REFRESHED column of "ceph orch ps" is blank for all daemons
• BZ - 2241825 - Change report frequency
• BZ - 2241907 - [rgw][s3select]: Connection broken error seen querying some csv objects downloaded from internet
• BZ - 2241945 - NVMeof Upgrade failed in a cluster
• BZ - 2242116 - [Ceph Dashboard]: Upgrade the angular to 15
• BZ - 2242117 - [Ceph Dashboard]: Fix the broken layout in dashboard landing page
• BZ - 2242126 - [ceph-dashboard] rgw naming improvements
• BZ - 2242987 - [rgw][indexless]: on Indexless placement, rgw daemon crashes with " ceph_assert(index.type == BucketIndexType::Normal)"
• BZ - 2242992 - While importing token provide an option to select host and port in which new rgw services needs to be deployed
• BZ - 2243074 - transition/versioning: With retain head object set as true, the current version metadata is not preserved
• BZ - 2243645 - [CephFS-Dashboard] - Volume Edit page is not listing the Hosts associated with the Filesystem
• BZ - 2244354 - Ceph deployment for dashboard is failing
• BZ - 2245145 - [rgw][s3select]: rgw going down executing query "select DATE_DIFF(SECOND, utcnow(),date_add(year,1111111111111111111, utcnow())) from s3object;"
• BZ - 2245575 - [rgw][s3select]: wrong output seen for query "select trim(LEADING '1' from '111abcdef111') from s3object;"
• BZ - 2246077 - Need to add the cmount_path in ganesha export block to fix the NFS memory consumption issue
• BZ - 2246453 - Export creation is failing with -- Failed to call create_export on underlying FSAL Ceph. Mandatory field, cmount_path is missing from block (FSAL)"
• BZ - 2246574 - [Ceph Common]: "ceph-diff-sorted" missing from package
• BZ - 2246909 - Post enabling the object gateway module in dashboard page will not refresh automatically, requires manual reload of the browser to disappear info panel
• BZ - 2247138 - rgw-restore-bucket-index does not restore some versioned instance entries (7.0)
• BZ - 2247174 - userspace cephfs client can crash when upgrading from RHCS 6 to 7
• BZ - 2247187 - mds: report clients laggy due laggy OSDs only after checking any OSD is laggy
• BZ - 2247214 - mgr/dashboard: add the cmount_path in ganesha export block to fix the NFS memory consumption issue
• BZ - 2247233 - mgr/dashboard: fix rgw multisite error when no rgw entity is present
• BZ - 2247237 - mgr/cephadm: default nvmeof cpumask + fixing tgt_cmd_extra_args quoting
• BZ - 2247306 - librgw: crash on startup (maybe with init_args non-empty)
• BZ - 2247616 - NVMeof GW deployment failing with 0.0.5 version
• BZ - 2247670 - [rgw-multisite] : fix for segmentation fault found in upstream multisite testing
• BZ - 2248084 - [NFS-Ganesha] Updating the ganesha export config is failing with "1 validation errors in block FSAL"
• BZ - 2248116 - [LC][Version]: Applying LC rule to delete expired current objects, does not generate delete-marker resulting in deletion of all the objects including non-current
• BZ - 2248157 - [rgw][s3select]: wrong output seen for query "select * from s3object[*].phonenumbers where _1.type is NOT null;"
• BZ - 2248862 - [rgw][s3select]: rgw going down executing query "select trim( leading 132140533849470 from _3 ) from s3object;"
• BZ - 2248866 - [rgw][s3select]: rgw going down executing query "select cast( trim( leading 132140533849470.72 from _3 ) as float) from s3object;"
• BZ - 2252272 - [Ceph Dashboard] Dashboard > Documentation link for rhcs 7.0 pointing to the wrong page
• BZ - 2253118 - Upgrade grafana version to 9.4.12

CVEs

(none)

References

(none)