There are two prerequisites that you must meet before you can use the Ceph Block Devices and the rbd command:

Before adding a block device to a node, create an image for it in the Ceph storage cluster. To create a block device image, execute the following command:

rbd create <image-name> --size <megabytes> --pool <pool-name>

For example, to create a 1GB image named data that stores information in a pool named stack, run:

$ rbd create data --size 1024 --pool stack

To list block devices in the rbd pool, execute the following (rbd is the default pool name):

rbd ls

To list block devices in a particular pool, execute the following, but replace {poolname} with the name of the pool:

rbd ls {poolname}

For example:

rbd ls swimmingpool

To retrieve information from a particular image, execute the following, but replace {image-name} with the name for the image:

rbd --image {image-name} info

For example:

rbd --image foo info

To retrieve information from an image within a pool, execute the following, but replace {image-name} with the name of the image and replace {pool-name} with the name of the pool:

rbd --image {image-name} -p {pool-name} info

For example:

rbd --image bar -p swimmingpool info

Ceph block device images are thin provisioned. They do not actually use any physical storage until you begin saving data to them. However, they do have a maximum capacity that you set with the --size option.

To increase or decrease the maximum size of a Ceph block device image:

rbd resize --image <image-name> --size <size>

To remove a block device, execute the following, but replace {image-name} with the name of the image you want to remove:

rbd rm {image-name}

For example:

rbd rm foo

To remove a block device from a pool, execute the following, but replace {image-name} with the name of the image to remove and replace {pool-name} with the name of the pool:

rbd rm {image-name} -p {pool-name}

For example:

rbd rm bar -p swimmingpool

You can enable or disable image features, such as fast-diff, exclusive-lock, object-map, or journaling, on already existing images.

To enable a feature:

rbd feature enable <pool-name>/<image-name> <feature-name>

To disable a feature:

rbd feature disable <pool-name>/<image-name> <feature-name>

Examples

rbd object-map rebuild <pool-name>/<image-name>

Ceph supports adding custom image metadata as key-value pairs. The pairs do not have any strict format.

Also, by using metadata, you can set the RBD configuration parameters for particular images. See Overriding the Default Configuration for Particular Images for details.

Use the rbd image-meta commands to work with metadata.

rbd image-meta set <pool-name>/<image-name> <key> <value>

Example

rbd image-meta remove <pool-name>/<image-name> <key>

Example

rbd image-meta get <pool-name>/<image-name> <key>

Example

rbd image-meta list <pool-name>/<image-name>

Example

rbd image-meta set <pool-name>/<image-name> conf_<parameter> <value>

Example

See Block Device Configuration Reference for a list of possible configuration options.

Use the rbd help command to display help for a particular rbd command and its subcommand:

rbd help <command> <subcommand>

Example

To display help for the snap list command:

$ rbd help snap list

When cephx is enabled (it is by default), you must specify a user name or ID and a path to the keyring containing the corresponding key for the user. You may also add the CEPH_ARGS environment variable to avoid re-entry of the following parameters:

rbd --id {user-ID} --keyring=/path/to/secret [commands]
rbd --name {username} --keyring=/path/to/secret [commands]

For example:

rbd --id admin --keyring=/etc/ceph/ceph.keyring [commands]
rbd --name client.admin --keyring=/etc/ceph/ceph.keyring [commands]

The following procedures demonstrate how to create, list, and remove snapshots using the rbd command on the command line.

rbd --pool {pool-name} snap create --snap {snap-name} {image-name}
rbd snap create {pool-name}/{image-name}@{snap-name}

rbd --pool rbd snap create --snap snapname foo
rbd snap create rbd/foo@snapname

rbd --pool {pool-name} snap ls {image-name}
rbd snap ls {pool-name}/{image-name}

rbd --pool rbd snap ls foo
rbd snap ls rbd/foo

rbd --pool {pool-name} snap rollback --snap {snap-name} {image-name}
rbd snap rollback {pool-name}/{image-name}@{snap-name}

rbd --pool rbd snap rollback --snap snapname foo
rbd snap rollback rbd/foo@snapname

rbd --pool <pool-name> snap rm --snap <snap-name> <image-name>
rbd snap rm <pool-name-/<image-name>@<snap-name>

$ rbd --pool rbd snap rm --snap snapname foo
$ rbd snap rm rbd/foo@snapname

rbd --pool {pool-name} snap purge {image-name}
rbd snap purge {pool-name}/{image-name}

rbd --pool rbd snap purge foo
rbd snap purge rbd/foo

rbd snap rename <pool-name>/<image-name>@<original-snapshot-name> <pool-name>/<image-name>@<new-snapshot-name>

$ rbd snap rename data/dataset@snap1 data/dataset@snap2

Ceph supports the ability to create many copy-on-write (COW) or copy-on-read (COR) clones of a block device snapshot. Snapshot layering enables Ceph block device clients to create images very quickly. For example, you might create a block device image with a Linux VM written to it; then, snapshot the image, protect the snapshot, and create as many clones as you like. A snapshot is read-only, so cloning a snapshot simplifies semantics—​making it possible to create clones rapidly.

Each cloned image (child) stores a reference to its parent image, which enables the cloned image to open the parent snapshot and read it. This reference is removed when the clone is flattened that is, when information from the snapshot is completely copied to the clone. For more information on flattening see Section 3.3.6, “Flattening Cloned Images”.

A clone of a snapshot behaves exactly like any other Ceph block device image. You can read to, write from, clone, and resize cloned images. There are no special restrictions with cloned images. However, the clone of a snapshot refers to the snapshot, so you MUST protect the snapshot before you clone it.

A clone of a snapshot can be a copy-on-write (COW) or copy-on-read (COR) clone. Copy-on-write (COW) is always enabled for clones while copy-on-read (COR) has to be enabled explicitly. Copy-on-write (COW) copies data from the parent to the clone when it writes to an unallocated object within the clone. Copy-on-read (COR) copies data from the parent to the clone when it reads from an unallocated object within the clone. Reading data from a clone will only read data from the parent if the object does not yet exist in the clone. Rados block device breaks up large images into multiple objects (defaults to 4 MB) and all copy-on-write (COW) and copy-on-read (COR) operations occur on a full object (that is writing 1 byte to a clone will result in a 4 MB object being read from the parent and written to the clone if the destination object does not already exist in the clone from a previous COW/COR operation).

Whether or not copy-on-read (COR) is enabled, any reads that cannot be satisfied by reading an underlying object from the clone will be rerouted to the parent. Since there is practically no limit to the number of parents (meaning that you can clone a clone), this reroute continues until an object is found or you hit the base parent image. If copy-on-read (COR) is enabled, any reads that fail to be satisfied directly from the clone result in a full object read from the parent and writing that data to the clone so that future reads of the same extent can be satisfied from the clone itself without the need of reading from the parent.

This is essentially an on-demand, object-by-object flatten operation. This is specially useful when the clone is in a high-latency connection away from it’s parent (parent in a different pool in another geographical location). Copy-on-read (COR) reduces the amortized latency of reads. The first few reads will have high latency because it will result in extra data being read from the parent (for example, you read 1 byte from the clone but now 4 MB has to be read from the parent and written to the clone), but all future reads will be served from the clone itself.

To create copy-on-read (COR) clones from snapshot you have to explicitly enable this feature by adding rbd_clone_copy_on_read = true under [global] or [client] section in your ceph.conf file.

3.3.1. Getting Started with Layering

Ceph block device layering is a simple process. You must have an image. You must create a snapshot of the image. You must protect the snapshot. Once you have performed these steps, you can begin cloning the snapshot.

The cloned image has a reference to the parent snapshot, and includes the pool ID, image ID and snapshot ID. The inclusion of the pool ID means that you may clone snapshots from one pool to images in another pool.

1. Image Template: A common use case for block device layering is to create a master image and a snapshot that serves as a template for clones. For example, a user may create an image for a RHEL7 distribution and create a snapshot for it. Periodically, the user may update the image and create a new snapshot (for example yum update, yum upgrade, followed by rbd snap create). As the image matures, the user can clone any one of the snapshots.
2. Extended Template: A more advanced use case includes extending a template image that provides more information than a base image. For example, a user may clone an image (for example, a VM template) and install other software (for example, a database, a content management system, an analytics system, and so on) and then snapshot the extended image, which itself may be updated just like the base image.
3. Template Pool: One way to use block device layering is to create a pool that contains master images that act as templates, and snapshots of those templates. You may then extend read-only privileges to users so that they may clone the snapshots without the ability to write or execute within the pool.
4. Image Migration/Recovery: One way to use block device layering is to migrate or recover data from one pool into another pool.

rbd --pool {pool-name} snap protect --image {image-name} --snap {snapshot-name}
rbd snap protect {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd snap protect --image my-image --snap my-snapshot
rbd snap protect rbd/my-image@my-snapshot

rbd --pool {pool-name} --image {parent-image} --snap {snap-name} --dest-pool {pool-name} --dest {child-image}
rbd clone {pool-name}/{parent-image}@{snap-name} {pool-name}/{child-image-name}

rbd clone rbd/my-image@my-snapshot rbd/new-image

rbd --pool {pool-name} snap unprotect --image {image-name} --snap {snapshot-name}
rbd snap unprotect {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd snap unprotect --image my-image --snap my-snapshot
rbd snap unprotect rbd/my-image@my-snapshot

rbd --pool {pool-name} children --image {image-name} --snap {snap-name}
rbd children {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd children --image my-image --snap my-snapshot
rbd children rbd/my-image@my-snapshot

rbd --pool <pool-name> flatten --image <image-name>
rbd flatten <pool-name>/<image-name>

$ rbd --pool rbd flatten --image my-image
$ rbd flatten rbd/my-image

You can enable the RBD journaling feature:

To enable journaling when creating an image, use the --image-feature option:

rbd create <image-name> --size <megabytes> --pool <pool-name> --image-feature <feature>

For example:

$ rbd create image-1 --size 1024 --pool pool-1 --image-feature exclusive-lock,journaling

To enable journaling on already existing images, use the rbd feature enable command:

rbd feature enable <pool-name>/<image-name> <feature-name>

For example:

$ rbd feature enable pool-1/image-1 exclusive-lock
$ rbd feature enable pool-1/image-1 journaling

To enable journaling on all new images by default, add the following setting to the Ceph configuration file:

rbd default features = 125

This chapter shows how to do the following tasks:

Execute the following commands on both peer clusters.

Enabling Mirroring on a Pool

To enable mirroring on a pool:

rbd mirror pool enable <pool-name> <mode>

Examples

To enable mirroring of the whole pool named data:

$ rbd mirror pool enable data pool

To enable image mode mirroring on the pool named stack:

$ rbd mirror pool enable stack image

See Mirroring Modes for details.

Disabling Mirroring on a Pool

To disable mirroring on a pool:

rbd mirror pool disable <pool-name>

Example

To disable mirroring of a pool named data:

$ rbd mirror pool disable data

Before disabling mirroring, remove the peer clusters. See Section 4.2, “Pool Configuration” for details.

rbd mirror pool peer add <pool-name> <client-name>@<cluster-name>

Example

To add the remote cluster as a peer to the local cluster or the other way around:

$ rbd --cluster local mirror pool peer add data client.remote@remote
$ rbd --cluster remote mirror pool peer add data client.local@local

Viewing Information about Peers

To view information about the peers:

rbd mirror pool info <pool-name>

Example

$ rbd mirror pool info data
Enabled: true
Peers:
  UUID                                 NAME        CLIENT
    786b42ea-97eb-4b16-95f4-867f02b67289 ceph-remote client.admin

Removing a Cluster Peer

To remove a mirroring peer cluster:

rbd mirror pool peer remove <pool-name> <peer-uuid>

Specify the pool name and the peer Universally Unique Identifier (UUID). To view the peer UUID, use the rbd mirror pool info command.

Example

$ rbd mirror pool peer remove data 55672766-c02b-4729-8567-f13a66893445

Getting Mirroring Status for a Pool

To get the mirroring pool summary:

rbd mirror pool status <pool-name>

Example

To get the status of the data pool:

$ rbd mirror pool status data
health: OK
images: 1 total

To output status details for every mirroring image in a pool, use the --verbose option.

This chapter shows how to do the following tasks:

Execute the following commands on a single cluster only.

Enabling Image Mirroring

To enable mirroring of a specific image:

Example

To enable mirroring for the image2 image in the data pool:

$ rbd mirror image enable data/image2

Disabling Image Mirroring

To disable mirroring for a specific image:

rbd mirror image disable <pool-name>/<image-name>

Example

To disable mirroring of the image2 image in the data pool:

$ rbd mirror image disable data/image2

Image Promotion and Demotion

To demote an image to non-primary:

rbd mirror image demote <pool-name>/<image-name>

Example

To demote the image2 image in the data pool:

$ rbd mirror image demote data/image2

To promote an image to primary:

rbd mirror image promote <pool-name>/<image-name>

Example

To promote the image2 image in the data pool:

$ rbd mirror image promote data/image2

See Section 4.6, “Recovering from a Disaster” for details.

Use the --force option to force promote a non-primary image:

$ rbd mirror image promote --force data/image2

Use forced promotion when the demotion cannot be propagated to the peer Ceph cluster, for example because of cluster failure or communication outage. See Failover After a Non-Orderly Shutdown for details.

Image Resynchronization

To request a resynchronization to the primary image:

rbd mirror image resync <pool-name>/<image-name>

Example

To request resynchronization of the image2 image in the data pool:

$ rbd mirror image resync data/image2

In case of an inconsistent state between the two peer clusters, the rbd-mirror daemon does not attempt to mirror the image that causing it. For details on fixing this issue, see Section 4.6, “Recovering from a Disaster”.

Getting Mirroring Status for a Single Image

To get status of a mirrored image:

rbd mirror image status <pool-name>/<image-name>

Example

To get the status of the image2 image in the data pool:

$ rbd mirror image status data/image2
image2:
  global_id:   2c928338-4a86-458b-9381-e68158da8970
  state:       up+replaying
  description: replaying, master_position=[object_number=6, tag_tid=2,
entry_tid=22598], mirror_position=[object_number=6, tag_tid=2,
entry_tid=29598], entries_behind_master=0
  last_update: 2016-04-28 18:47:39

One-way mirroring implies that a primary image in one cluster gets replicated in a secondary cluster. In the secondary or remote cluster, the replicated image is non-primary; that is, block device clients cannot write to the image.

The following procedures assume:

There are two ways to configure block device mirroring:

The following procedures assume that:

The following procedures show how to fail over to the mirrored data on the secondary cluster (named remote) after the primary one (named local) terminated, and how to Failback. The shutdown can be:

If the shutdown is non-orderly, the Failback procedure requires resynchronizing the image.

The procedures assume that you have successfully configured either:

When updating a cluster using Ceph Block Device mirroring with an asynchronous update, follow the installation instruction for the update. Then, restart the Ceph Block Device instances.

To mount a block device image, first return a list of the images.

To do so, execute the following:

rbd list

Use rbd to map an image name to a kernel module. You must specify the image name, the pool name and the user name. rbd will load RBD kernel module on your behalf if it is not already loaded.

To do so, execute the following:

sudo rbd map {image-name} --pool {pool-name} --id {user-name}

For example:

sudo rbd map --pool rbd myimage --id admin

If you use cephx authentication, you must also specify a secret. It may come from a keyring or a file containing the secret.

To do so, execute the following:

sudo rbd map --pool rbd myimage --id admin --keyring /path/to/keyring
sudo rbd map --pool rbd myimage --id admin --keyfile /path/to/file

To show block device images mapped to kernel modules with the rbd command, specify the showmapped option.

To do so, execute the following:

rbd showmapped

To unmap a block device image with the rbd command, specify the unmap option and the device name (by convention the same as the block device image name).

To do so, execute the following:

sudo rbd unmap /dev/rbd/{poolname}/{imagename}

For example:

sudo rbd unmap /dev/rbd/rbd/foo

It is possible to override the default settings for creating an image. Ceph will create images with format 2 and no striping.

The user space implementation of the Ceph block device (that is, librbd) cannot take advantage of the Linux page cache, so it includes its own in-memory caching, called RBD caching. RBD caching behaves just like well-behaved hard disk caching. When the OS sends a barrier or a flush request, all dirty data is written to the OSDs. This means that using write-back caching is just as safe as using a well-behaved physical hard disk with a VM that properly sends flushes (that is, Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) algorithm, and in write-back mode it can coalesce contiguous requests for better throughput.

Ceph supports write-back caching for RBD. To enable it, add rbd cache = true to the [client] section of your ceph.conf file. By default librbd does not perform any caching. Writes and reads go directly to the storage cluster, and writes return only when the data is on disk on all replicas. With caching enabled, writes return immediately, unless there are more than rbd cache max dirty unflushed bytes. In this case, the write triggers writeback and blocks until enough bytes are flushed.

Ceph supports write-through caching for RBD. You can set the size of the cache, and you can set targets and limits to switch from write-back caching to write through caching. To enable write-through mode, set rbd cache max dirty to 0. This means writes return only when the data is on disk on all replicas, but reads may come from the cache. The cache is in memory on the client, and each RBD image has its own. Since the cache is local to the client, there is no coherency if there are others accessing the image. Running GFS or OCFS on top of RBD will not work with caching enabled.

The ceph.conf file settings for RBD should be set in the [client] section of your configuration file. The settings include:

RBD supports read-ahead/prefetching to optimize small, sequential reads. This should normally be handled by the guest OS in the case of a VM, but boot loaders may not issue efficient reads. Read-ahead is automatically disabled if caching is disabled.