How does a large number of objects in a RHCS/Ceph pool affect the filestore merge and split rate, as well as the cluster performance?

Solution In Progress - Updated -

Issue

  • A large number of slow requests and high disk utilisation while reading/writing data to pools in the Ceph cluster.

  • Several OSDs were dying with either failed or wrongly marked me down messages, as well as with heartbeat related stack traces.

  • Slow requests evenly distributed throughout all the SATA disks in the cluster. This indicates the problem is cluster-wide and not isolated to any specific disk, host, or rack.

  • Listing the PG directory took well over 30 seconds to complete due to the high number of files.

Environment

  • Red Hat Ceph Storage 1.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.