Chapter 4. Server and rack solutions

Hardware vendors have responded to the enthusiasm around Ceph by providing both optimized server-level and rack-level solution SKUs. Validated through joint testing with Red Hat, these solutions offer predictable price-to-performance ratios for Ceph deployments, with a convenient modular approach to expand Ceph storage for specific workloads.

Typical rack-level solutions include:

  • Network switching: Redundant network switching interconnects the cluster and provides access to clients. A minimum of one network switch is recommended. For redundancy purpose, two network switches are used, with each switch partitioned to support two network segments.
  • Ceph MON nodes: The Ceph monitor is a datastore for the health of the entire cluster, and contains the cluster log. A minimum of three monitor nodes are strongly recommended for a cluster quorum in production.
  • Ceph OSD hosts: Ceph OSD hosts house the storage capacity for the cluster, with one or more OSDs running per individual storage device if the device is HDD or SSD. For NVME device, Red Hat recommends to have two or more OSDs running per individual storage device. OSD hosts are selected and configured differently depending on both workload optimization and the data devices installed: HDDs, SSDs, or NVMe SSDs.
  • Red Hat Ceph Storage: Many vendors provide a capacity-based subscription for Red Hat Ceph Storage bundled with both server and rack-level solution SKUs.
Note

Red Hat recommends to review the Red Hat Ceph Storage:Supported Configurations article prior to committing to any server and rack solution. Contact Red Hat support for any additional assistance.

IOPS-optimized solutions

With the growing use of flash storage, organizations increasingly host IOPS-intensive workloads on Ceph storage clusters to let them emulate high-performance public cloud solutions with private cloud storage. These workloads commonly involve structured data from MySQL-, MariaDB-, or PostgreSQL-based applications.

Typical servers include the following elements:

  • CPU: 6 cores per NVMe SSD, assuming a 2 GHz CPU.
  • RAM: 16 GB baseline, plus 5 GB per OSD.
  • Networking: 10 Gigabit Ethernet, GbE, per 2 OSDs.
  • OSD media: High-performance, high-endurance enterprise NVMe SSDs.
  • OSDs: Two per NVMe SSD.
  • Bluestore WAL/DB: High-performance, high-endurance enterprise NVMe SSD, co-located with OSDs.
  • Controller: Native PCIe bus.
Note

For Non-NVMe SSDs, for CPU, use two cores per SSD OSD.

Table 4.1. Solutions SKUs for IOPS-optimized Ceph Workloads, by cluster size.

VendorSmall (250TB)Medium (1PB)Large (2PB+)

SuperMicro [a]

SYS-5038MR-OSD006P

N/A

N/A

Throughput-optimized Solutions

Throughput-optimized Ceph solutions are usually centered around semi-structured or unstructured data. Large-block sequential I/O is typical. Storage media on OSD hosts is commonly HDDs with write journals on SSD-based volumes.

Typical server elements include:

  • CPU: 0.5 cores per HDD, assuming a 2 GHz CPU.
  • RAM: 16 GB baseline, plus 5 GB per OSD.
  • Networking: 10 GbE per 12 OSDs each for client and cluster-facing networks.
  • OSD media: 7,200 RPM enterprise HDDs.
  • OSDs: One per HDD.
  • Bluestore WAL/DB: High-endurance, high-performance enterprise serial-attached SCSI (SAS) or NVMe SSDs.
  • Host bus adapter (HBA): Just a bunch of disks (JBOD).

Several vendors provide pre-configured server and rack-level solutions for throughput-optimized Ceph workloads. Red Hat has conducted extensive testing and evaluation of servers from Supermicro and Quanta Cloud Technologies (QCT).

Table 4.2. Rack-level SKUs for Ceph OSDs, MONs, and top-of-rack (TOR) switches.

VendorSmall (250TB)Medium (1PB)Large (2PB+)

SuperMicro [a]

SRS-42E112-Ceph-03

SRS-42E136-Ceph-03

SRS-42E136-Ceph-03

Table 4.3. Individual OSD Servers

VendorSmall (250TB)Medium (1PB)Large (2PB+)

SuperMicro [a]

SSG-6028R-OSD072P

SSG-6048-OSD216P

SSG-6048-OSD216P

QCT [a]

QxStor RCT-200

QxStor RCT-400

QxStor RCT-400

Table 4.4. Additional Servers Configurable for Throughput-optmized Ceph OSD Workloads.

VendorSmall (250TB)Medium (1PB)Large (2PB+)

Dell

PowerEdge R730XD [a]

DSS 7000 [b], twin node

DSS 7000, twin node

Cisco

UCS C240 M4

UCS C3260 [c]

UCS C3260 [d]

Lenovo

System x3650 M5

System x3650 M5

N/A

[d] See UCS C3260 for details

Cost and capacity-optimized solutions

Cost- and capacity-optimized solutions typically focus on higher capacity, or longer archival scenarios. Data can be either semi-structured or unstructured. Workloads include media archives, big data analytics archives, and machine image backups. Large-block sequential I/O is typical. For greater cost effectiveness, OSDs are usually hosted on HDDs with Ceph write journals co-located on the HDDs.

Solutions typically include the following elements:

  • CPU: 0.5 cores per HDD, assuming a 2 GHz CPU.
  • RAM: 16 GB baseline, plus 5 GB per OSD.
  • Networking: 10 GbE per 12 OSDs (each for client- and cluster-facing networks).
  • OSD media: 7,200 RPM enterprise HDDs.
  • OSDs: One per HDD.
  • Bluestore WAL/DB: Co-located on the HDD.
  • HBA: JBOD.

Supermicro and QCT provide pre-configured server and rack-level solution SKUs for cost- and capacity-focused Ceph workloads.

Table 4.5. Pre-configured Rack-level SKUs for Cost- and Capacity-optimized Workloads

VendorSmall (250TB)Medium (1PB)Large (2PB+)

SuperMicro [a]

N/A

SRS-42E136-Ceph-03

SRS-42E172-Ceph-03

Table 4.6. Pre-configured Server-level SKUs for Cost- and Capacity-optimized Workloads

VendorSmall (250TB)Medium (1PB)Large (2PB+)

SuperMicro [a]

N/A

SSG-6048R-OSD216P [a]

SSD-6048R-OSD360P

QCT

N/A

QxStor RCC-400 [a]

QxStor RCC-400 [a]

Table 4.7. Additional Servers Configurable for Cost- and Capacity-optimized Workloads

VendorSmall (250TB)Medium (1PB)Large (2PB+)

Dell

N/A

DSS 7000, twin node

DSS 7000, twin node

Cisco

N/A

UCS C3260

UCS C3260

Lenovo

N/A

System x3650 M5

N/A