- Red Hat Ceph Storage
- RGW Multisite environment
- Executing the
ceph health detailshows a large omap warning in the bucket index pool
HEALTH_WARN 3 large omap objects LARGE_OMAP_OBJECTS 3 large omap objects 3 large objects found in pool 'zz1.rgw.buckets.index' Search the cluster log for 'Large omap object found' for more details.
- Bucket ID in the ceph.log belongs to deleted bucket listed under stale instances
If the bilog count or size is greater than the configuration parameter osd_deep_scrub_large_omap_object_key_threshold and osd_deep_scrub_large_omap_object_value_sum_threshold which defaults to 2000000 and 1_G respectively
Execute the bilog trim
$ radosgw-admin bilog trim --bucket="bucket-name" --bucket-id="bucket-id"
Execute the deep-scrub onto the PG manually where that large omap object is located ( In the ceph.log file )
$ ceph osd pg deep-scrub <pgid>
- In the multisite environment, it is by design that the deleted buckets are not removed automatically
- If the deleted bucket index id shows a large omap warning, either the sync has not completed successfully or the bilogs are not getting trimmed
- Large OMAP warning in the ceph.log start with ".dir" indicating that it belongs to the index pool
2021-9-05 06:54:25.093024 osd.144 (osd.144) 592 : cluster [WRN] Large omap object found. Object: 89:039c6a78:::.dir.e9ff35e0-e817-5363-9d1c-eefe85926cab.32068365.1.1:head PG: 89.1e5639c0 (89.40) Key count: 649368 Size (bytes): 128983232
- In the above log entry, bucket ID is e9ff35e0-e817-5363-9d1c-eefe85926cab.32068365.1
Get the bucket name from the metadata instance list and confirm that it is not listed under the metadata list or bucket stats rather listed under the stale instances list.
Get the bucket name
$ radosgw-admin metadata list --metadata-key bucket.instance | grep -i "bucket-id"
Match the bucket ID with the live or active buckets list
$ radosgw-admin metadata list --metadata-key bucket | grep -i "bucket-name"
$ radosgw-admin bucket stats | grep -i "bucket-name"
At the end, match the bucket ID with the stale bucket instances list
$ radosgw-admin reshard stale-instances list --yes-i-really-mean-it | grep -i "bucket-id"
Check the bucket and cluster sync status for the multisite sync
Cluster sync status
$ radosgw-admin sync status
Bucket sync status
$ radosgw-admin bucket sync status --bucket="bucket-name" --bucket-id="bucket-id"
Check the bilog entries for the bucket showing large omap warning
$ radosgw-admin bilog list --bucket-id="bucket-id" --bucket-name="bucket-name" --max-entries=600000 | grep -c op_id
$ rados -p zz1.rgw.index listomapkeys .dir.e9ff35e0-e817-5363-9d1c-eefe85926cab.32068365.1.1 | grep -i "0_000" | wc -l
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.