To use Ceph Block Device commands on your client node, you will have to meet some prerequisites.

ceph-deploy --overwrite-conf config push <client-node>

Before you can add a block device to a node, you must create an image for it in the Ceph Storage Cluster first. To create a block device image, execute the following:

rbd create {image-name} --size {megabytes} --pool {pool-name}

For example, to create a 1GB image named foo that stores information in a pool named swimmingpool, execute the following:

rbd create foo --size 1024
rbd create bar --size 1024 --pool swimmingpool

To list block devices in the rbd pool, execute the following (i.e., rbd is the default pool name):

rbd ls

To list block devices in a particular pool, execute the following, but replace {poolname} with the name of the pool:

rbd ls {poolname}

For example:

rbd ls swimmingpool

To retrieve information from a particular image, execute the following, but replace {image-name} with the name for the image:

rbd --image {image-name} info

For example:

rbd --image foo info

To retrieve information from an image within a pool, execute the following, but replace {image-name} with the name of the image and replace {pool-name} with the name of the pool:

rbd --image {image-name} -p {pool-name} info

For example:

rbd --image bar -p swimmingpool info

Ceph Block Device images are thin provisioned. They don’t actually use any physical storage until you begin saving data to them. However, they do have a maximum capacity that you set with the --size option. If you want to increase (or decrease) the maximum size of a Ceph Block Device image, execute the following:

rbd resize --image foo --size 2048

To remove a block device, execute the following, but replace {image-name} with the name of the image you want to remove:

rbd rm {image-name}

For example:

rbd rm foo

To remove a block device from a pool, execute the following, but replace {image-name} with the name of the image to remove and replace {pool-name} with the name of the pool:

rbd rm {image-name} -p {pool-name}

For example:

rbd rm bar -p swimmingpool

When cephx is enabled (it is by default), you must specify a user name or ID and a path to the keyring containing the corresponding key for the user. You may also add the CEPH_ARGS environment variable to avoid re-entry of the following parameters:

rbd --id {user-ID} --keyring=/path/to/secret [commands]
rbd --name {username} --keyring=/path/to/secret [commands]

For example:

rbd --id admin --keyring=/etc/ceph/ceph.keyring [commands]
rbd --name client.admin --keyring=/etc/ceph/ceph.keyring [commands]

The following procedures demonstrate how to create, list, and remove snapshots using the rbd command on the command line.

rbd --pool {pool-name} snap create --snap {snap-name} {image-name}
rbd snap create {pool-name}/{image-name}@{snap-name}

rbd --pool rbd snap create --snap snapname foo
rbd snap create rbd/foo@snapname

rbd --pool {pool-name} snap ls {image-name}
rbd snap ls {pool-name}/{image-name}

rbd --pool rbd snap ls foo
rbd snap ls rbd/foo

rbd --pool {pool-name} snap rollback --snap {snap-name} {image-name}
rbd snap rollback {pool-name}/{image-name}@{snap-name}

rbd --pool rbd snap rollback --snap snapname foo
rbd snap rollback rbd/foo@snapname

rbd --pool {pool-name} snap rm --snap {snap-name} {image-name}
rbd snap rm {pool-name}/{image-name}@{snap-name}

rbd --pool rbd snap rm --snap snapname foo
rbd snap rm rbd/foo@snapname

rbd --pool {pool-name} snap purge {image-name}
rbd snap purge {pool-name}/{image-name}

rbd --pool rbd snap purge foo
rbd snap purge rbd/foo

Ceph supports the ability to create many copy-on-write (COW) clones of a block device shapshot. Snapshot layering enables Ceph block device clients to create images very quickly. For example, you might create a block device image with a Linux VM written to it; then, snapshot the image, protect the snapshot, and create as many copy-on-write clones as you like. A snapshot is read-only, so cloning a snapshot simplifies semantics—​making it possible to create clones rapidly.

Each cloned image (child) stores a reference to its parent image, which enables the cloned image to open the parent snapshot and read it.

A COW clone of a snapshot behaves exactly like any other Ceph block device image. You can read to, write from, clone, and resize cloned images. There are no special restrictions with cloned images. However, the copy-on-write clone of a snapshot refers to the snapshot, so you MUST protect the snapshot before you clone it. The following diagram depicts the process.

9.1. Getting Started with Layering

Ceph block device layering is a simple process. You must have an image. You must create a snapshot of the image. You must protect the snapshot. Once you have performed these steps, you can begin cloning the snapshot.

The cloned image has a reference to the parent snapshot, and includes the pool ID, image ID and snapshot ID. The inclusion of the pool ID means that you may clone snapshots from one pool to images in another pool.

1. Image Template: A common use case for block device layering is to create a a master image and a snapshot that serves as a template for clones. For example, a user may create an image for a RHEL7 distribution and create a snapshot for it. Periodically, the user may update the image and create a new snapshot (e.g.sudo yum update, sudo yum upgrade, followed by rbd snap create). As the image matures, the user can clone any one of the snapshots.
2. Extended Template: A more advanced use case includes extending a template image that provides more information than a base image. For example, a user may clone an image (e.g., a VM template) and install other software (e.g., a database, a content management system, an analytics system, etc.) and then snapshot the extended image, which itself may be updated just like the base image.
3. Template Pool: One way to use block device layering is to create a pool that contains master images that act as templates, and snapshots of those templates. You may then extend read-only privileges to users so that they may clone the snapshots without the ability to write or execute within the pool.
4. Image Migration/Recovery: One way to use block device layering is to migrate or recover data from one pool into another pool.

rbd --pool {pool-name} snap protect --image {image-name} --snap {snapshot-name}
rbd snap protect {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd snap protect --image my-image --snap my-snapshot
rbd snap protect rbd/my-image@my-snapshot

rbd --pool {pool-name} --image {parent-image} --snap {snap-name} --dest-pool {pool-name} --dest {child-image}
rbd clone {pool-name}/{parent-image}@{snap-name} {pool-name}/{child-image-name}

rbd clone rbd/my-image@my-snapshot rbd/new-image

rbd --pool {pool-name} snap unprotect --image {image-name} --snap {snapshot-name}
rbd snap unprotect {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd snap unprotect --image my-image --snap my-snapshot
rbd snap unprotect rbd/my-image@my-snapshot

rbd --pool {pool-name} children --image {image-name} --snap {snap-name}
rbd children {pool-name}/{image-name}@{snapshot-name}

rbd --pool rbd children --image my-image --snap my-snapshot
rbd children rbd/my-image@my-snapshot

rbd --pool {pool-name} flatten --image {image-name}
rbd flatten {pool-name}/{image-name}

rbd --pool rbd flatten --image my-image
rbd flatten rbd/my-image

The user space implementation of the Ceph block device (i.e, librbd) cannot take advantage of the Linux page cache, so it includes its own in-memory caching, called RBD caching. RBD caching behaves just like well-behaved hard disk caching. When the OS sends a barrier or a flush request, all dirty data is written to the OSDs. This means that using write-back caching is just as safe as using a well-behaved physical hard disk with a VM that properly sends flushes (i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) algorithm, and in write-back mode it can coalesce contiguous requests for better throughput.

Ceph supports write-back caching for RBD. To enable it, add rbd cache = true to the [client] section of your ceph.conf file. By default librbd does not perform any caching. Writes and reads go directly to the storage cluster, and writes return only when the data is on disk on all replicas. With caching enabled, writes return immediately, unless there are more than rbd cache max dirty unflushed bytes. In this case, the write triggers writeback and blocks until enough bytes are flushed.

Ceph supports write-through caching for RBD. You can set the size of the cache, and you can set targets and limits to switch from write-back caching to write through caching. To enable write-through mode, set rbd cache max dirty to 0. This means writes return only when the data is on disk on all replicas, but reads may come from the cache. The cache is in memory on the client, and each RBD image has its own. Since the cache is local to the client, there’s no coherency if there are others accessing the image. Running GFS or OCFS on top of RBD will not work with caching enabled.

The ceph.conf file settings for RBD should be set in the [client] section of your configuration file. The settings include:

rbd cache

rbd cache size

rbd cache max dirty

rbd cache target dirty

rbd cache max dirty age

rbd cache writethrough until flush

RBD supports read-ahead/prefetching to optimize small, sequential reads. This should normally be handled by the guest OS in the case of a VM, but boot loaders may not issue efficient reads. Read-ahead is automatically disabled if caching is disabled.

rbd readahead trigger requests

rbd readahead max bytes

rbd readahead disable after bytes

To use rbd, you must first connect to RADOS and open an IO context:

cluster = rados.Rados(conffile='my_ceph.conf')
cluster.connect()
ioctx = cluster.open_ioctx('mypool')

Then you instantiate an :class:rbd.RBD object, which you use to create the image:

rbd_inst = rbd.RBD()
size = 4 * 1024**3  # 4 GiB
rbd_inst.create(ioctx, 'myimage', size)

To perform I/O on the image, you instantiate an :class:rbd.Image object:

image = rbd.Image(ioctx, 'myimage')
data = 'foo' * 200
image.write(data, 0)

This writes foo to the first 600 bytes of the image. Note that data cannot be :type:unicode - Librbd does not know how to deal with characters wider than a :c:type:char.

In the end, you’ll close the image, the IO context and the connection to RADOS:

image.close()
ioctx.close()
cluster.shutdown()

To be safe, each of these calls would need to be in a separate :finally block:

cluster = rados.Rados(conffile='my_ceph_conf')
try:
    ioctx = cluster.open_ioctx('my_pool')
    try:
        rbd_inst = rbd.RBD()
        size = 4 * 1024**3  # 4 GiB
        rbd_inst.create(ioctx, 'myimage', size)
        image = rbd.Image(ioctx, 'myimage')
        try:
            data = 'foo' * 200
            image.write(data, 0)
        finally:
            image.close()
    finally:
        ioctx.close()
finally:
    cluster.shutdown()

This can be cumbersome, so the Rados, Ioctx, and Image classes can be used as context managers that close/shutdown automatically. Using them as context managers, the above example becomes:

with rados.Rados(conffile='my_ceph.conf') as cluster:
    with cluster.open_ioctx('mypool') as ioctx:
        rbd_inst = rbd.RBD()
        size = 4 * 1024**3  # 4 GiB
        rbd_inst.create(ioctx, 'myimage', size)
        with rbd.Image(ioctx, 'myimage') as image:
            data = 'foo' * 200
            image.write(data, 0)

To mount a block device image, first return a list of the images.

To do so, execute the following:

rbd list

Use rbd to map an image name to a kernel module. You must specify the image name, the pool name and the user name. rbd will load RBD kernel module on your behalf if it’s not already loaded.

To do so, execute the following:

sudo rbd map {image-name} --pool {pool-name} --id {user-name}

For example:

sudo rbd map --pool rbd myimage --id admin

If you use cephx authentication, you must also specify a secret. It may come from a keyring or a file containing the secret.

To do so, execute the following:

sudo rbd map --pool rbd myimage --id admin --keyring /path/to/keyring
sudo rbd map --pool rbd myimage --id admin --keyfile /path/to/file

To show block device images mapped to kernel modules with the rbd command, specify the showmapped option.

To do so, execute the following:

rbd showmapped

To unmap a block device image with the rbd command, specify the unmap option and the device name (i.e. by convention the same as the block device image name).

To do so, execute the following:

sudo rbd unmap /dev/rbd/{poolname}/{imagename}

For example:

sudo rbd unmap /dev/rbd/rbd/foo