Chapter 6. Cache Tiering (Tech Preview)

Setting up a backing storage pool typically involves one of two scenarios:

In the standard storage scenario, you can setup a CRUSH ruleset to establish the failure domain (e.g., osd, host, chassis, rack, row, etc.). Ceph OSD Daemons perform optimally when all storage drives in the ruleset are of the same size, speed (both RPMs and throughput) and type. See CRUSH Maps for details on creating a ruleset. Once you have created a ruleset, create a backing storage pool.

In the erasure coding scenario, the pool creation arguments will generate the appropriate ruleset automatically. See Create a Pool for details.

In subsequent examples, we will refer to the backing storage pool as cold-storage.

Setting up a cache pool follows the same procedure as the standard storage scenario, but with this difference: the drives for the cache tier are typically high performance drives that reside in their own servers and have their own ruleset. When setting up a ruleset, it should take account of the hosts that have the high performance drives while omitting the hosts that don’t. See CRUSH Storage Strategy Examples.

In subsequent examples, we will refer to the cache pool as hot-storage and the backing pool as cold-storage.

For cache tier configuration and default values, see Pools - Set Pool Values.

Setting up a cache tier involves associating a backing storage pool with a cache pool :

ceph osd tier add {storagepool} {cachepool}

For example :

ceph osd tier add cold-storage hot-storage

To set the cache mode, execute the following:

ceph osd tier cache-mode {cachepool} {cache-mode}

For example:

ceph osd tier cache-mode hot-storage writeback

Writeback cache tiers overlay the backing storage tier, so they require one additional step: you must direct all client traffic from the storage pool to the cache pool. To direct client traffic directly to the cache pool, execute the following:

ceph osd tier set-overlay {storagepool} {cachepool}

For example:

ceph osd tier set-overlay cold-storage hot-storage

Cache tiers have several configuration options. You may set cache tier configuration options with the following usage:

ceph osd pool set {cachepool} {key} {value}

See Pools - Set Pool Values for details.

Ceph’s production cache tiers use a Bloom Filter for the hit_set_type:

ceph osd pool set {cachepool} hit_set_type bloom

For example:

ceph osd pool set hot-storage hit_set_type bloom

The hit_set_count and hit_set_period define how much time each HitSet should cover, and how many such HitSets to store. Currently there is minimal benefit for hit_set_count > 1 since the agent does not yet act intelligently on that information. :

ceph osd pool set {cachepool} hit_set_count 1
ceph osd pool set {cachepool} hit_set_period 3600
ceph osd pool set {cachepool} target_max_bytes 1000000000000

Binning accesses over time allows Ceph to determine whether a Ceph client accessed an object at least once, or more than once over a time period ("age" vs "temperature").

The cache tiering agent performs two main functions:

ceph osd pool set {cachepool} cache_target_dirty_ratio {0.0..1.0}

ceph osd pool set hot-storage cache_target_dirty_ratio 0.4

ceph osd pool set {cachepool} cache_target_full_ratio {0.0..1.0}

ceph osd pool set hot-storage cache_target_full_ratio 0.8

ceph osd pool set {cachepool} target_max_bytes {#bytes}

ceph osd pool hot-storage target_max_bytes 1000000000000

ceph osd pool set {cachepool} target_max_objects {#objects}

ceph osd pool set hot-storage target_max_objects 1000000

You can specify the minimum age of an object before the cache tiering agent flushes a recently modified (or dirty) object to the backing storage pool:

ceph osd pool set {cachepool} cache_min_flush_age {#seconds}

For example, to flush modified (or dirty) objects after 10 minutes, execute the following:

ceph osd pool set hot-storage cache_min_flush_age 600

You can specify the minimum age of an object before it will be evicted from the cache tier:

ceph osd pool {cache-tier} cache_min_evict_age {#seconds}

For example, to evict objects after 30 minutes, execute the following:

ceph osd pool set hot-storage cache_min_evict_age 1800

Since a read-only cache does not have modified data, you can disable and remove it without losing any recent changes to objects in the cache.

Since a writeback cache may have modified data, you must take steps to ensure that you do not lose any recent changes to objects in the cache before you disable and remove it.