Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 2. Overview

Red Hat Ceph Storage v1.3 is the second release of Red Hat Ceph Storage. New features for Ceph Storage include:

2.1. Packaging

For organizations that require highly secure clusters, Red Hat Ceph Storage ships with an ISO-based installation so that you can deploy Ceph without a connection to the internet. For organizations that allow the Ceph cluster to connect to the internet, Red Hat Ceph Storage supports a CDN-based installation (RHEL only).

For RHEL 7 both ISO-based and CDN-based installation are available.

For Ubuntu 14.04 currently only ISO-based installation is available. The first point release of Red Hat Ceph Storage v1.3 for Ubuntu 14.04 will introduce an online repository based installation.

Red Hat Ceph Storage v1.3 for RHEL 7 ships with two Stock Keeping Units (SKUs).

  • Red Hat Ceph Storage for Management Nodes: The repositories for this SKU provide access to the installer, Calamari and Ceph monitors. You may use this SKU on up to six physical nodes.
  • Red Hat Ceph Storage: The repository for this SKU provides access to OSDs. You will need one SKU for each node containing Ceph OSDs.

For CDN-based installations, you will need to attach pools for for these SKUs. See the Installation Guide for details.

Red Hat Ceph Storage v1.3 for RHEL 7 has the following repositories:

  • rhel-7-server-rhceph-1.3-calamari-rpms contains the Calamari repository.
  • rhel-7-server-rhceph-1.3-installer-rpms contains the ceph-deploy repository.
  • rhel-7-server-rhceph-1.3-mon-rpms contains the ceph-mon daemon.
  • rhel-7-server-rhceph-1.3-osd-rpms contains the ceph-osd daemon.
  • rhel-7-server-rhceph-1.3-tools-rpms contains the ceph CLI tools, Ceph Block Device kernel for RHEL 7.1 and higher, and the Ceph Object Gateway.

You will need to enable these repositories on various hosts. For details about installation on RHEL 7, see RHCS v1.3 Installation Guide for RHEL (x86_64).

For details about installation on Ubuntu 14.04, see RHCS v1.3 Installation Guide for Ubuntu (x86_64).

2.2. Ceph Storage Cluster

The Ceph Storage Cluster has a number of new features and improvements.

  • Monitor Performance: Ceph monitors now perform writes to the local data store asynchronously, improving overall responsiveness.
  • Cache Tiering (Tech Preview): Cache tiering has some of the same overhead as the underlying storage tier, so it performs best under certain conditions. A series of changes have been made in the cache tiering code that improve performance and reduce latency; namely, objects are not promoted into the cache tier by a single read; instead, they must be found to be sufficiently hot before getting promoted to the cache tier.
  • New Administrator Commands: The ceph osd df command shows pertinent details on OSD disk utilizations. The ceph pg ls ... command makes it much simpler to query PG states while diagnosing cluster issues.
  • Local Recovery Codes (Tech Preview): the OSDs now support an erasure-coding scheme that stores some additional data blocks to reduce the IO required to recover from single OSD failures.
  • Degraded vs Misplaced: The Ceph health reports from ceph -s and related commands now make a distinction between data that is degraded (there are fewer than the desired number of copies) and data that is misplaced (stored in the wrong location in the cluster). The distinction is important because the latter does not compromise data safety.
  • Recovery Tools: The ceph-objectstore-tool allow you to mount an offline OSD disk, retrieve PGs and objects and manipulate them for debugging and repair purposes. Red Hat Ceph Storage support personnel are the heaviest users of this tool. Consult Red Hat Ceph Storage support before using the ceph-objecstore-tool.
  • CRUSH Improvements: We have added a new straw2 bucket algorithm that reduces the amount of data migration required when changes are made to the cluster.
  • OSD SSD Optimization: Ceph Storage v1.3 provides some optimization which results in less CPU overhead per operation and thus more operations. This improvement is relevant for fast hardware such as SSDs. If you experienced CPU bound operations with SSDs before, you should see an improvement. However, this will not address network bottlenecks.
  • Time-scheduled Scrubbing: Ceph Storage v1.3 supports osd_scrub_begin_hour and osd_scrub_end_hour time ranges as allowable hours for scrubbing. Ceph ignores these settings if the OSD exceeds osd_scrub_max_interval.

Erasure-coding and cache tiering are tech previews only and are not supported for production clusters.

2.3. Ceph Block Device

  • Mandatory Exclusive Locks: The mandatory locking framework (disabled by default) adds additional safeguards to prevent multiple clients from using the same image simultaneously. See the rbd --help interface for additional usage and the Ceph Architecture Guide for architecture details.
  • Copy-on-Read Cloning: Copy-on-read for image clones improves performance for some workloads. For example, when used with OpenStack the copy happens when you read data, and thereby happens a bit faster, reducing flattening time and allowing graduating to a parent clone. Copy-on-read is disabled by default, but you may enable it by setting rbd_clone_copy_on_read=true in your Ceph configuration.
  • Object Maps: Ceph Block Device now has an object map function that tracks which parts of the image are actually allocated (i.e., block devices are thin provisioned, so this shows the index of objects that actually exist). Object maps are valuable with clones that get objects form a parent, as they improve performance for clones when re-sizing and importing, exporting or flattening. Object maps are off by default, but you can enable them in your Ceph configuration by specifying rbd default format = 2 and rbd default features = X, where X is the sum of the feature bits.
  • Read-ahead: Ceph Block Device supports read ahead, which provides a small difference in improvement for virtio (e.g., 10%), but it doubles the improvement for IDE.
  • Allocation Hinting: The allocation hints are to prevent fragmentation on the filesystem beneath the OSD. The block device knows the size of its objects, so it sends that size as an allocation hint with write operations so the OSD reserves that amount of space for the object. With additional writes, the object will be sequential when it is fully written in the filesystem. This prevents performance degradation from fragmentation. Allocation hinting is on by default.
  • Cache Hinting: Ceph Block Device supports cache hinting, which makes more efficient use of the client side cache or cache tiering by making import/export slightly more efficient in Red Hat Ceph Storage v1.3.

2.4. Ceph Object Gateway

  • Civetweb Installation: The ceph-deploy tool now has a new ceph-deploy rgw create <HOST> command that quickly deploys an instance of the S3/Swift gateway using the embedded Civetweb server (defaulting to port 7480). The new installation method dramatically simplifies installation and configuration compared to Apache and FastCGI. Presently, it only supports HTTP. To use HTTPS, you may use a proxy server.
  • S3 API Object Versioning: Instead of deleting previous versions of S3 objects, the gateway will maintain a history of object versions.
  • Bucket Sharding: When buckets contain an extraordinary number of objects (e.g., 100k-1M+), bucket index performance degraded in previous releases. Bucket sharding dramatically improves performance in those scenarios.
  • Swift API Placement Policies: The Swift API now allows you to create a bucket and specify a placement pool key (e.g., mapping a bucket and its object to high performance pools, such as SSD backed pools).