Chapter 3. Optimize workload performance domains

One of the key benefits of Ceph storage is the ability to support different types of workloads within the same cluster using Ceph performance domains. Dramatically different hardware configurations can be associated with each performance domain. Ceph system administrators can deploy storage pools on the appropriate performance domain, providing applications with storage tailored to specific performance and cost profiles. Selecting appropriately sized and optimized servers for these performance domains is an essential aspect of designing a Red Hat Ceph Storage cluster.

The following lists provide the criteria Red Hat uses to identify optimal Red Hat Ceph Storage cluster configurations on storage servers. These categories are provided as general guidelines for hardware purchases and configuration decisions, and can be adjusted to satisfy unique workload blends. Actual hardware configurations chosen will vary depending on specific workload mix and vendor capabilities.

IOPS optimized

An IOPS-optimized storage cluster typically has the following properties:

  • Lowest cost per IOPS.
  • Highest IOPS per GB.
  • 99th percentile latency consistency.

Typically uses for an IOPS-optimized storage cluster are:

  • Typically block storage.
  • 3x replication for hard disk drives (HDDs) or 2x replication for solid state drives (SSDs).
  • MySQL on OpenStack clouds.

Throughput optimized

A throughput-optimized storage cluster typically has the following properties:

  • Lowest cost per MBps (throughput).
  • Highest MBps per TB.
  • Highest MBps per BTU.
  • Highest MBps per Watt.
  • 97th percentile latency consistency.

Typically uses for an throughput-optimized storage cluster are:

  • Block or object storage.
  • 3x replication.
  • Active performance storage for video, audio, and images.
  • Streaming media.

Cost and capacity optimized

A cost- and capacity-optimized storage cluster typically has the following properties:

  • Lowest cost per TB.
  • Lowest BTU per TB.
  • Lowest Watts required per TB.

Typically uses for an cost- and capacity-optimized storage cluster are:

  • Typically object storage.
  • Erasure coding common for maximizing usable capacity
  • Object archive.
  • Video, audio, and image object repositories.

How performance domains work

To the Ceph client interface that reads and writes data, a Ceph storage cluster appears as a simple pool where the client stores data. However, the storage cluster performs many complex operations in a manner that is completely transparent to the client interface. Ceph clients and Ceph object storage daemons (Ceph OSDs, or simply OSDs) both use the controlled replication under scalable hashing (CRUSH) algorithm for storage and retrieval of objects. OSDs run on OSD hosts—the storage servers within the cluster.

A CRUSH map describes a topography of cluster resources, and the map exists both on client nodes as well as Ceph Monitor (MON) nodes within the cluster. Ceph clients and Ceph OSDs both use the CRUSH map and the CRUSH algorithm. Ceph clients communicate directly with OSDs, eliminating a centralized object lookup and a potential performance bottleneck. With awareness of the CRUSH map and communication with their peers, OSDs can handle replication, backfilling, and recovery—allowing for dynamic failure recovery.

Ceph uses the CRUSH map to implement failure domains. Ceph also uses the CRUSH map to implement performance domains, which simply take the performance profile of the underlying hardware into consideration. The CRUSH map describes how Ceph stores data, and it is implemented as a simple hierarchy (acyclic graph) and a ruleset. The CRUSH map can support multiple hierarchies to separate one type of hardware performance profile from another. In RHCS 2 and earlier, performance domains reside in separate CRUSH hierarchies. In RHCS 3, Ceph implements performance domains with device "classes".

The following examples describe performance domains.

  • Hard disk drives (HDDs) are typically appropriate for cost- and capacity-focused workloads.
  • Throughput-sensitive workloads typically use HDDs with Ceph write journals on solid state drives (SSDs).
  • IOPS-intensive workloads such as MySQL and MariaDB often use SSDs.

All of these performance domains can coexist in a Ceph storage cluster.