Chapter 9. BlueStore

1. Add a new OSD node to the [osds] section in Ansible inventory file, by default located at /etc/ansible/hosts.

   [osds]
   node1
   node2
   node3
   <hostname>

   Replace:

   • <hostname> with the name of the OSD node

   For example:

   [osds]
   node1
   node2
   node3
   node4

2. Navigate to the /usr/share/ceph-ansible/ directory.

   [user@admin ~]$ cd /usr/share/ceph-ansible

3. Create the host_vars directory.

   [root@admin ceph-ansible] mkdir host_vars

4. Create the configuration file for the newly added OSD in host_vars.

   [root@admin ceph-ansible] touch host_vars/<hostname>.yml

   Replace:

   • <hostname> with the host name of the newly added OSD

   For example:

   [root@admin ceph-ansible] touch host_vars/node4.yml

5. Add the following setting to the newly created file:

   osd_objectstore: bluestore

   Note

   To use BlueStore for all OSDs, add osd_objectstore:bluestore to the group_vars/all.yml file.

6. Optional. If you want to store the block.wal and block.db partitions on dedicated devices, edit the host_vars/<hostname>.yml file as follows.

   1. To use dedicated devices for block.wal:

      osd_scenario: non-collocated
      
      bluestore_wal_devices:
         - <device>
         - <device>

      Replace:

      • <device> with the path to the device

      For example:

      osd_scenario: non-collocated
      
      bluestore_wal_devices:
         - /dev/sdf
         - /dev/sdg

   2. To use dedicated devices for block.db:

      osd_scenario: non-collocated
      
      dedicated_devices:
         - <device>
         - <device>

      Replace:

      • <device> with the path to the device

      For example:

      osd_scenario: non-collocated
      
      dedicated_devices:
         - /dev/sdh
         - /dev/sdi

      Note

      If you use the osd_scenario: collocated parameter, the block.wal and block.db partitions will use the same device as specified with the devices parameter. For details, see the Installing a Red Hat Ceph Storage Cluster section in the Red Hat Ceph Storage 3 l Installation Guide for Red Hat Enterprise Linux or Ubuntu.

      Note

      To use BlueStore for all OSDs, add the aforementioned parameters to the group_vars/osds.yml file.

   3. To override the block.db and block.wal default size in the group_vars/all.yml file:

      ceph_conf_overrides:
        osd:
          bluestore_block_db_size: <value>
          bluestore_block_wal_size: <value>

      Replace:

      • <value> with the size in bytes.

      For example:

      ceph_conf_overrides:
        osd:
          bluestore_block_db_size: 14336000000
          bluestore_block_wal_size: 2048000000

7. To configure LVM based BlueStore OSDs, use osd_scenario: lvm in host_vars/<hostname>.yml:

   osd_scenario: lvm
   lvm_volumes:
     - data: <datalv>
       data_vg: <datavg>

   Replace:

   • <datalv> with the data logical volume name
   • <datavg> with the data logical volume group name

   For example:

   osd_scenario: lvm
   lvm_volumes:
     - data: data-lv1
       data_vg: vg1

8. Optional. If you want to store the block.wal and block.db on dedicated logical volumes, edit the host_vars/<hostname>.yml file as follows:

   osd_scenario: lvm
   lvm_volumes:
     - data: <datalv>
       wal: <wallv>
       wal_vg: <vg>
       db: <dblv>
       db_vg: <vg>

   Replace:

   • <datalv> with the logical volume where the data should be contained
   • <wallv> with the logical volume where the write-ahead-log should be contained
   • <vg> with the volume group the WAL and/or DB device LVs are on
   • <dblv> with the logical volume the BlueStore internal metadata should be contained

   For example:

   osd_scenario: lvm
   lvm_volumes:
     - data: data-lv3
       wal: wal-lv1
       wal_vg: vg3
       db: db-lv3
       db_vg: vg3

   Note

   When using lvm_volumes: with osd_objectstore: bluestore the lvm_volumes YAML dictionary must contain at least data. When defining wal or db, it must have both the LV name and VG name (db and wal are not required). This allows for four combinations: just data, data and wal, data and wal and db, or data and db. Data can be a raw device, lv or partition. The wal and db can be a lv or partition. When specifying a raw device or partition ceph-volume will put logical volumes on top of them.

   Note

   Currently, ceph-ansible does not create the volume groups or the logical volumes. This must be done before running the Anisble playbook.

9. Open and edit the group_vars/all.yml file, and uncomment the osd_memory_target option. Adjust the value on how much memory you want the OSD to consume.

   Note

   The default value for the osd_memory_target option is 4000000000, which is 4 GB. This option pins the BlueStore cache in memory.

   Important

   The osd_memory_target option only applies to BlueStore-backed OSDs.

10. Use the ansible-playbook:

    [user@admin ceph-ansible]$ ansible-playbook site.yml

11. From a Monitor node, verify that the new OSD has been successfully added:

    [root@monitor ~]# ceph osd tree

Procedure

1. Optional: If redeploying an existing OSD node, copy the admin keyring from /etc/ceph/ on the Ceph Monitor node to the node from which you want to remove the OSD.

2. Optional: If redeploying an existing OSD node, use the shrink-osd.yml Ansible playbook to remove the OSD from the cluster.

   ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=OSD_ID

   Example

   [admin@admin ceph-ansible]$ ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=1

3. If redeploying an existing OSD node, wipe the OSD drives and reinstall the OS.
4. Prepare the node for OSD provisioning using Ansible. Such as, enabling {storage-product} repositories, adding an Ansible user, and enabling password-less SSH login.

5. Add the bluestore_min_alloc_size to the ceph_conf_overrides section of the group_vars/all.yml Ansible playbook:

   ceph_conf_overrides:
     osd:
       bluestore_min_alloc_size: 4096

6. If deploying a new node, add it to the Ansible inventory file, normally /etc/ansible/hosts:

   [osds]
   OSD_NODE_NAME

   Example

   [osds]
   osd1 devices="[ '/dev/sdb' ]"

7. Provision the OSD node using Ansible:

   ansible-playbook -v site.yml -l OSD_NODE_NAME

   Example

   [admin@admin ceph-ansible]$ ansible-playbook -v site.yml -l osd1

8. After the playbook finishes, verify the setting using the ceph daemon command:

   ceph daemon OSD.ID config get bluestore_min_alloc_size

   Example

   [root@osd1 ~]# ceph daemon osd.1 config get bluestore_min_alloc_size
   {
       "bluestore_min_alloc_size": "4096"
   }

   You can see bluestore_min_alloc_size is set to 4096 bytes, which is equivalent to 4 KB.