Chapter 9. BlueStore

Starting with Red Hat Ceph Storage 4, BlueStore is the default object store for the OSD daemons. The earlier object store, FileStore, requires a file system on top of raw block devices. Objects are then written to the file system. BlueStore does not require an initial file system, because BlueStore puts objects directly on the block device.

Important

BlueStore provides a high-performance backend for OSD daemons in a production environment. By default, BlueStore is configured to be self-tuning. If you determine that your environment performs better with BlueStore tuned manually, please contact Red Hat support and share the details of your configuration to help us improve the auto-tuning capability. Red Hat looks forward to your feedback and appreciates your recommendations.

9.1. Ceph BlueStore

The following are some of the main features of using BlueStore:

Direct management of storage devices
BlueStore consumes raw block devices or partitions. This avoids any intervening layers of abstraction, such as local file systems like XFS, that might limit performance or add complexity.
Metadata management with RocksDB
BlueStore uses the RocksDB’ key-value database to manage internal metadata, such as the mapping from object names to block locations on a disk.
Full data and metadata checksumming
By default all data and metadata written to BlueStore is protected by one or more checksums. No data or metadata are read from disk or returned to the user without verification.
Efficient copy-on-write
The Ceph Block Device and Ceph File System snapshots rely on a copy-on-write clone mechanism that is implemented efficiently in BlueStore. This results in efficient I/O both for regular snapshots and for erasure coded pools which rely on cloning to implement efficient two-phase commits.
No large double-writes
BlueStore first writes any new data to unallocated space on a block device, and then commits a RocksDB transaction that updates the object metadata to reference the new region of the disk. Only when the write operation is below a configurable size threshold, it falls back to a write-ahead journaling scheme, similar to how FileStore operates.
Multi-device support

BlueStore can use multiple block devices for storing different data. For example: Hard Disk Drive (HDD) for the data, Solid-state Drive (SSD) for metadata, Non-volatile Memory (NVM) or Non-volatile random-access memory (NVRAM) or persistent memory for the RocksDB write-ahead log (WAL). See Ceph BlueStore devices for details.

Note

The ceph-disk utility does not yet provision multiple devices. To use multiple devices, OSDs must be set up manually.

Efficient block device usage
Because BlueStore does not use any file system, it minimizes the need to clear the storage device cache.

9.2. Ceph BlueStore devices

This section explains what block devices the BlueStore back end uses.

BlueStore manages either one, two, or three storage devices.

  • Primary
  • WAL
  • DB

In the simplest case, BlueStore consumes a single (primary) storage device. The storage device is partitioned into two parts that contain:

  • OSD metadata: A small partition formatted with XFS that contains basic metadata for the OSD. This data directory includes information about the OSD, such as its identifier, which cluster it belongs to, and its private keyring.
  • Data: A large partition occupying the rest of the device that is managed directly by BlueStore and that contains all of the OSD data. This primary device is identified by a block symbolic link in the data directory.

You can also use two additional devices:

  • A WAL (write-ahead-log) device: A device that stores BlueStore internal journal or write-ahead log. It is identified by the block.wal symbolic link in the data directory. Consider using a WAL device only if the device is faster than the primary device. For example, when the WAL device uses an SSD disk and the primary devices uses an HDD disk.
  • A DB device: A device that stores BlueStore internal metadata. The embedded RocksDB database puts as much metadata as it can on the DB device instead of on the primary device to improve performance. If the DB device is full, it starts adding metadata to the primary device. Consider using a DB device only if the device is faster than the primary device.
Warning

If you have only a less than a gigabyte storage available on fast devices. Red Hat recommends using it as a WAL device. If you have more fast devices available, consider using it as a DB device. The BlueStore journal is always placed on the fastest device, so using a DB device provides the same benefit that the WAL device while also allows for storing additional metadata.

9.3. Ceph BlueStore caching

The BlueStore cache is a collection of buffers that, depending on configuration, can be populated with data as the OSD daemon does reading from or writing to the disk. By default in Red Hat Ceph Storage, BlueStore will cache on reads, but not writes. This is because the bluestore_default_buffered_write option is set to false to avoid potential overhead associated with cache eviction.

If the bluestore_default_buffered_write option is set to true, data is written to the buffer first, and then committed to disk. Afterwards, a write acknowledgement is sent to the client, allowing subsequent reads faster access to the data already in cache, until that data is evicted.

Read-heavy workloads will not see an immediate benefit from BlueStore caching. As more reading is done, the cache will grow over time and subsequent reads will see an improvement in performance. How fast the cache populates depends on the BlueStore block and database disk type, and the client’s workload requirements.

Important

Please contact Red Hat support before enabling the bluestore_default_buffered_write option.

9.4. Sizing considerations for Ceph BlueStore

When mixing traditional and solid state drives using BlueStore OSDs, it is important to size the RocksDB logical volume (block.db) appropriately. Red Hat recommends that the RocksDB logical volume be no less than 4% of the block size with object, file and mixed workloads. Red Hat supports 1% of the BlueStore block size with RocksDB and OpenStack block workloads. For example, if the block size is 1 TB for an object workload, then at a minimum, create a 40 GB RocksDB logical volume.

When not mixing drive types, there is no requirement to have a separate RocksDB logical volume. BlueStore will automatically manage the sizing of RocksDB.

BlueStore’s cache memory is used for the key-value pair metadata for RocksDB, BlueStore metadata and object data.

Note

The BlueStore cache memory values are in addition to the memory footprint already being consumed by the OSD.

9.5. Adding Ceph BlueStore OSDs

This section describes how to install a new Ceph OSD node with the BlueStore back end object store.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the node.

Procedure

  1. Add a new OSD node to the [osds] section in Ansible inventory file, by default located at /etc/ansible/hosts.

    [osds]
    node1
    node2
    node3
    HOST_NAME

    Replace:

    • HOST_NAME with the name of the OSD node

    Example

    [osds]
    node1
    node2
    node3
    node4

  2. Navigate to the /usr/share/ceph-ansible/ directory.

    [user@admin ~]$ cd /usr/share/ceph-ansible
  3. Create the host_vars directory.

    [root@admin ceph-ansible] mkdir host_vars
  4. Create the configuration file for the newly added OSD in host_vars.

    [root@admin ceph-ansible] touch host_vars/HOST_NAME.yml

    Replace:

    • HOST_NAME with the host name of the newly added OSD

    Example

    [root@admin ceph-ansible] touch host_vars/node4.yml

  5. Add the following setting to the newly created file:

    osd_objectstore: bluestore
    Note

    To use BlueStore for all OSDs, add osd_objectstore:bluestore to the group_vars/all.yml file.

  6. Configure the BlueStore OSDs, in host_vars/HOST_NAME.yml:

    lvm_volumes:
      - data: DATALV
        data_vg: DATAVG

    Replace:

    • DATALV with the data logical volume name
    • DATAVG with the data logical volume group name

    Example

    lvm_volumes:
      - data: data-lv1
        data_vg: vg1

  7. Optional. If you want to store the block.wal and block.db on dedicated logical volumes, edit the host_vars/HOST_NAME.yml file as follows:

    lvm_volumes:
      - data: DATALV
        wal: WALLV
        wal_vg: VG
        db: DBLV
        db_vg: VG

    Replace:

    • DATALV with the logical volume where the data should be contained
    • WALLV with the logical volume where the write-ahead-log should be contained
    • VG with the volume group the WAL and/or DB device LVs are on
    • DBLV with the logical volume the BlueStore internal metadata should be contained

    Example

    lvm_volumes:
      - data: data-lv3
        wal: wal-lv1
        wal_vg: vg3
        db: db-lv3
        db_vg: vg3

    Note

    When using lvm_volumes: with osd_objectstore: bluestore the lvm_volumes YAML dictionary must contain at least data. When defining wal or db, it must have both the LV name and VG name (db and wal are not required). This allows for four combinations: just data, data and wal, data and wal and db, or data and db. Data can be a raw device, lv or partition. The wal and db can be a lv or partition. When specifying a raw device or partition ceph-volume will put logical volumes on top of them.

    Note

    Currently, ceph-ansible does not create the volume groups or the logical volumes. This must be done before running the Anisble playbook.

  8. Open and edit the group_vars/all.yml file, and uncomment the osd_memory_target option. Adjust the value on how much memory you want the OSD to consume.

    Note

    The default value for the osd_memory_target option is 4000000000, which is 4 GB. This option pins the BlueStore cache in memory.

    Important

    The osd_memory_target option only applies to BlueStore-backed OSDs.

  9. Run the following Ansible playbook:

    [user@admin ceph-ansible]$ ansible-playbook site.yml
  10. From a Ceph Monitor node, verify that the new OSD has been successfully added:

    [root@mon ~]# ceph osd tree

9.6. Tuning Ceph BlueStore for small writes

In BlueStore, the raw partition is allocated and managed in chunks of bluestore_min_alloc_size. By default, bluestore_min_alloc_size is 64 KB for HDDs, and 16 KB for SSDs. The unwritten area in each chunk is filled with zeroes when it is written to the raw partition. This can lead to wasted unused space when not properly sized for your workload, for example when writing small objects.

It is best practice to set bluestore_min_alloc_size to match the smallest write so this can write amplification penalty can be avoided.

For example, if your client writes 4 KB objects frequently, use ceph-ansible to configure the following setting on OSD nodes:

bluestore_min_alloc_size = 4096

Note

The settings bluestore_min_alloc_size_ssd and bluestore_min_alloc_size_hdd are specific to SSDs and HDDs, respectively, but setting them is not necessary because setting bluestore_min_alloc_size overrides them.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • New servers that can be freshly provisioned as OSD nodes, or:
  • OSD nodes that can be redeployed.
  • The admin keyring for the Ceph Monitor node, if you are redeploying an existing Ceph OSD node.

Procedure

  1. Optional: If redeploying an existing OSD node, use the shrink-osd.yml Ansible playbook to remove the OSD from the cluster.

    ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=OSD_ID

    Example

    [admin@admin ceph-ansible]$ ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=1

  2. If redeploying an existing OSD node, wipe the OSD drives and reinstall the OS.
  3. Prepare the node for OSD provisioning using Ansible. Examples of preparation tasks include enabling Red Hat Ceph Storage repositories, adding an Ansible user, and enabling password-less SSH login.
  4. Add the bluestore_min_alloc_size to the ceph_conf_overrides section of the group_vars/all.yml Ansible playbook:

    ceph_conf_overrides:
      osd:
        bluestore_min_alloc_size: 4096
  5. If deploying a new node, add it to the Ansible inventory file, normally /etc/ansible/hosts:

    [osds]
    OSD_NODE_NAME

    Example

    [osds]
    osd1 devices="[ '/dev/sdb' ]"

  6. If redeploying an existing OSD, copy the admin keyring file in the Ceph Monitor node to the node where you want to deploy the OSD.
  7. Provision the OSD node using Ansible:

    ansible-playbook -v site.yml -l OSD_NODE_NAME

    Example

    [admin@admin ceph-ansible]$ ansible-playbook -v site.yml -l osd1

  8. After the playbook finishes, verify the setting using the ceph daemon command:

    ceph daemon OSD.ID config get bluestore_min_alloc_size

    Example

    [root@osd1 ~]# ceph daemon osd.1 config get bluestore_min_alloc_size
    {
        "bluestore_min_alloc_size": "4096"
    }

    You can see bluestore_min_alloc_size is set to 4096 bytes, which is equivalent to 4 KiB.

Additional Resources

9.7. The BlueStore fragmentation tool

As a storage administrator, you will want to periodically check the fragmentation level of your BlueStore OSDs. You can check fragmentation levels with one simple command for offline or online OSDs.

9.7.1. Prerequisites

  • A running Red Hat Ceph Storage 3.3 or higher storage cluster.
  • BlueStore OSDs.

9.7.2. What is the BlueStore fragmentation tool?

For BlueStore OSDs, the free space gets fragmented over time on the underlying storage device. Some fragmentation is normal, but when there is excessive fragmentation this causes poor performance.

The BlueStore fragmentation tool generates a score on the fragmentation level of the BlueStore OSD. This fragmentation score is given as a range, 0 through 1. A score of 0 means no fragmentation, and a score of 1 means severe fragmentation.

Table 9.1. Fragmentation scores' meaning

ScoreFragmentation Amount

0.0 - 0.4

None to tiny fragmentation.

0.4 - 0.7

Small and acceptable fragmentation.

0.7 - 0.9

Considerable, but safe fragmentation.

0.9 - 1.0

Severe fragmentation and that causes performance issues.

Important

If you have severe fragmentation, and need some help in resolving the issue, contact Red Hat Support.

9.7.3. Checking for fragmentation

Checking the fragmentation level of BlueStore OSDs can be done either online or offline.

Prerequisites

  • A running Red Hat Ceph Storage 3.3 or higher storage cluster.
  • BlueStore OSDs.

Online BlueStore fragmentation score

  1. Inspect a running BlueStore OSD process:

    1. Simple report:

      Syntax

      ceph daemon OSD_ID bluestore allocator score block

      Example

      [root@osd ~]# ceph daemon osd.123 bluestore allocator score block

    2. A more detailed report:

      Syntax

      ceph daemon OSD_ID bluestore allocator dump block

      Example

      [root@osd ~]# ceph daemon osd.123 bluestore allocator dump block

Offline BlueStore fragmentation score

  1. Inspect a non-running BlueStore OSD process:

    1. Simple report:

      Syntax

      ceph-bluestore-tool --path PATH_TO_OSD_DATA_DIRECTORY --allocator block free-score

      Example

      [root@osd ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score

    2. A more detailed report:

      Syntax

      ceph-bluestore-tool --path PATH_TO_OSD_DATA_DIRECTORY --allocator block free-dump

      Example

      [root@osd ~]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump

Additional Resources

9.8. How to Migrate the Object Store from FileStore to BlueStore

As a storage administrator, you can migrate from the traditional object store, FileStore, to the new object store, BlueStore.

9.8.1. Prerequisites

  • A healthy and running Red Hat Ceph Storage cluster.

9.8.2. Migrating from FileStore to BlueStore

BlueStore improves performance and robustness, compared to the traditional FileStore. A single Red Hat Ceph Storage cluster can contain a mix of both FileStore and BlueStore devices.

Converting an individual OSD cannot be done in place, or in isolation. The conversion process will rely either on the storage cluster’s normal replication and healing process or tools and strategies that copy OSD content from an old (FileStore) device to a new (BlueStore) device. There are two approach to migrate from FileStore to BlueStore.

First Approach

The first approach is to mark out each device in turn, wait for the data to replicate across the storage cluster, reprovision the OSD, and mark it back "in" again. Here are the advantages and disadvantage to this approach:

Advantages
  • Simple.
  • Can be done on a device-by-device basis.
  • No spare devices or nodes are required.
Disadvantages
  • Copying data over the network happens twice.

    Note

    One copy to some other OSD in the storage cluster, allowing you to maintain the desired number of replicas, and then another copy back to the reprovisioned BlueStore OSD.

Second Approach

The second approach is doing a whole node replacement. You need to have an empty node that has no data.

There are two ways to do this: * Starting with a new, empty node that is not part of the storage cluster. * By offloading data from an existing node in the storage cluster.

Advantages
  • Data is copied over the network only once.
  • Converts an entire node’s OSDs at once.
  • Can parallelize to converting multiple nodes at a time.
  • No spare devices are required on each node.
Disadvantages
  • A spare node is required.
  • An entire node’s worth of OSDs will be migrating data at a time. This is like likely to impact overall cluster performance.
  • All migrated data still makes one full hop over the network.

9.8.3. Migrating from FileStore to BlueStore using Ansible

Migrating from FileStore to BlueStore using Ansible will shrinks and redeploys all OSDs on the node. The Ansible playbook does a capacity check before starting the migration. The ceph-volume utility then redeploys the OSDs.

Prerequisites

  • A healthy and running Red Hat Ceph Storage 4 cluster.
  • The ansible user account for use with the Ansible application.

Procedure

  1. Log in as the ansible user on the Ansible administration node.
  2. Edit the group_vars/osd.yml file, add and set the following options:

    nb_retry_wait_osd_up: 50
    delay_wait_osd_up: 30
  3. Run the following Ansible playbook:

    Syntax

    ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit OSD_NODE_TO_MIGRATE

    Example

    [ansible@admin ~]$ ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit osd1

  4. Wait for the migration to complete before starting on the next OSD node in the storage cluster.

9.8.4. Migrating from FileStore to BlueStore using the mark out and replace approach

The simplest approach to migrate from FileStore to BlueStore is to mark out each device in turn, wait for the data to replicate across the storage cluster, reprovision the OSD, and mark it back "in" again.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root access to the node.

Procedure

Replace the variable OSD_ID below with the ODS identification number.

  1. Find a FileStore OSD to replace.

    1. Get the OSD identification number:

      [root@ceph-client ~]# ceph osd tree
    2. Identify whether an OSD is using FileStore or BlueStore:

      Syntax

      ceph osd metadata OSD_ID | grep osd_objectstore

      Example

      [root@ceph-client ~]# ceph osd metadata 0 | grep osd_objectstore
          "osd_objectstore": "filestore",

    3. To view the current count of FileStore devices versus BlueStore devices:

      [root@ceph-client ~]# ceph osd count-metadata osd_objectstore
  2. Mark the FileStore OSD out:

    ceph osd out OSD_ID
  3. Wait for the data to migrate off the OSD:

    while ! ceph osd safe-to-destroy OSD_ID ; do sleep 60 ; done
  4. Stop the OSD:

    systemctl stop ceph-osd@OSD_ID
  5. Capture which device this OSD is using:

    mount | grep /var/lib/ceph/osd/ceph-OSD_ID
  6. Unmount the OSD:

    umount /var/lib/ceph/osd/ceph-OSD_ID
  7. Destroy the OSD data, using the value from step 5 as DEVICE:

    ceph-volume lvm zap DEVICE
    Important

    Be EXTREMELY CAREFUL as this will destroy the contents of the device. Be certain the data on the device is not needed, that is the storage cluster is healthy, before proceeding.``

    Note

    If the OSD is encrypted, then the unmount the osd-lockbox and remove the encryption before zapping the OSD using dmsetup remove.

    Note

    If the OSD contains logical volumes, then use the --destroy option on the ceph-volume lvm zap command.

  8. Make the storage cluster aware that the OSD has been destroyed:

    [root@ceph-client ~]# ceph osd destroy OSD_ID --yes-i-really-mean-it
  9. Reprovision the OSD as a BlueStore OSD, using DEVICE from step 5, and the same OSD_ID:

    [root@ceph-client ~]# ceph-volume lvm create --bluestore --data DEVICE --osd-id OSD_ID
  10. Repeat this procedure.

    Note

    The refilling of the new BlueStore OSD can happen concurrently with the draining of the next FileStore OSD, as long as you ensure the storage cluster is HEALTH_OK before destroying any OSDs. Failure to do so will reduce the redundancy of your data and increase the risk of, or the potentially of data loss.

9.8.5. Migrating from FileStore to BlueStore using the whole node replacement approach

Migrating from FileStore to BlueStore can be done on a node-by-node basis by transferring each stored copy of the data only once. This migration can be done with a spare node in the storage cluster, or having the sufficient free space to evacuate an entire node from the storage cluster in order to use it as a spare. Ideally, the node must have roughly the same capacity as the other nodes you will be migrating.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root access to the node.
  • An empty node that has no data.

Procedure

  • Replace the variable NEWNODE below with the new node name.
  • Replace the variable EXISTING_NODE_TO_CONVERT below with the node name already existing in the storage cluster.
  • Replace the variable OSD_ID below with the OSD identification number.

    1. Using a new node that is not in the storage cluster. For using an existing node already in the storage cluster, skip to step 3.

      1. Add the node to the CRUSH hierarchy:

        [root@mon ~]# ceph osd crush add-bucket NEWNODE node
        Important

        Do not attach it to the root.

      2. Install the Ceph software packages:

        [root@mon ~]# yum install ceph-osd
        Note

        Copy the Ceph configuration file, by default /etc/ceph/ceph.conf, and keyrings to the new node.

    2. Skip to step 5.
    3. If you are using an existing node already in the storage cluster, use the following command:

      [root@mon ~]# ceph osd crush unlink EXISTING_NODE_TO_CONVERT default
      Note

      Where default is the immediate ancestor in the CRUSH map.

    4. Skip to step 8.
    5. Provision new BlueStore OSDs for all devices:

      [root@mon ~]# ceph-volume lvm create --bluestore --data /dev/DEVICE
    6. Verify that OSDs joined the cluster:

      [root@mon ~]# ceph osd tree

      You should see the new node name with all of the OSDs underneath the node name, but the node must not be nested underneath any other node in hierarchy.

      Example

      [root@mon ~]# ceph osd tree
      ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
      -5             0 node newnode
      10   ssd 1.00000     osd.10        up  1.00000 1.00000
      11   ssd 1.00000     osd.11        up  1.00000 1.00000
      12   ssd 1.00000     osd.12        up  1.00000 1.00000
      -1       3.00000 root default
      -2       3.00000     node oldnode1
      0   ssd 1.00000         osd.0     up  1.00000 1.00000
      1   ssd 1.00000         osd.1     up  1.00000 1.00000
      2   ssd 1.00000         osd.2     up  1.00000 1.00000

    7. Swap the new node into the old node’s position in the cluster:

      [root@mon ~]# ceph osd crush swap-bucket NEWNODE EXISTING_NODE_TO_CONVERT

      At this point, all data on the EXISTING_NODE_TO_CONVERT will start migrating to OSDs on the NEWNODE.

      Note

      If there is a difference in the total capacity of the old and new nodes you might also see some data migrate to or from other nodes in the storage cluster, but as long as the nodes are similarly sized this will be a relatively small amount of data.

    8. Wait for data migration to complete:

      while ! ceph osd safe-to-destroy $(ceph osd ls-tree EXISTING_NODE_TO_CONVERT); do sleep 60 ; done
    9. Log into the EXISTING_NODE_TO_CONVERT, stop and unmount all old OSDs on the now-empty EXISTING_NODE_TO_CONVERT:

      [root@mon ~]# systemctl stop ceph-osd@OSD_ID
      [root@mon ~]# umount /var/lib/ceph/osd/ceph-OSD_ID
    10. Destroy and purge the old OSDs:

      for osd in ceph osd ls-tree EXISTING_NODE_TO_CONVERT; do ceph osd purge $osd --yes-i-really-mean-it ; done
    11. Wipe the old OSD devices. This requires you do identify which devices are to be wiped manually. Do the following command for each device:

      [root@mon ~]# ceph-volume lvm zap DEVICE
      Important

      Be EXTREMELY CAREFUL as this will destroy the contents of the device. Be certain the data on the device is not needed, that is the storage cluster is healthy, before proceeding.

      Note

      If the OSD is encrypted, then the unmount the osd-lockbox and remove the encryption before zapping the OSD using dmsetup remove.

      Note

      If the OSD contains logical volumes, then use the --destroy option on the ceph-volume lvm zap command.

    12. Use the now-empty old node as the new node, and repeat the process.