Chapter 5. Customizing the Ceph Storage cluster
Director deploys containerized Red Hat Ceph Storage using a default configuration. You can customize Ceph Storage by overriding the default settings.
Prerequistes
To deploy containerized Ceph Storage you must include the /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml
file during overcloud deployment. This environment file defines the following resources:
-
CephAnsibleDisksConfig
- This resource maps the Ceph Storage node disk layout. For more information, see Section 5.3, “Mapping the Ceph Storage node disk layout”. -
CephConfigOverrides
- This resource applies all other custom settings to your Ceph Storage cluster.
Use these resources to override any defaults that the director sets for containerized Ceph Storage.
Procedure
Enable the Red Hat Ceph Storage 4 Tools repository:
$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
Install the
ceph-ansible
package on the undercloud:$ sudo dnf install ceph-ansible
To customize your Ceph Storage cluster, define custom parameters in a new environment file, for example,
/home/stack/templates/ceph-config.yaml
. You can apply Ceph Storage cluster settings with the following syntax in theparameter_defaults
section of your environment file:parameter_defaults: CephConfigOverrides: section: KEY:VALUE
NoteYou can apply the
CephConfigOverrides
parameter to the[global]
section of theceph.conf
file, as well as any other section, such as[osd]
,[mon]
, and[client]
. If you specify a section, thekey:value
data goes into the specified section. If you do not specify a section, the data goes into the[global]
section by default. For information about Ceph Storage configuration, customization, and supported parameters, see Red Hat Ceph Storage Configuration Guide.Replace
KEY
andVALUE
with the Ceph cluster settings that you want to apply. For example, in theglobal
section,max_open_files
is theKEY
and131072
is the correspondingVALUE
:parameter_defaults: CephConfigOverrides: global: max_open_files: 131072 osd: osd_scrub_during_recovery: false
This configuration results in the following settings defined in the configuration file of your Ceph cluster:
[global] max_open_files = 131072 [osd] osd_scrub_during_recovery = false
5.1. Setting ceph-ansible group variables
The ceph-ansible
tool is a playbook used to install and manage Ceph Storage clusters.
The ceph-ansible
tool has a group_vars
directory that defines configuration options and the default settings for those options. Use the group_vars
directory to set Ceph Storage parameters.
For information about the group_vars
directory, see Installing a Red Hat Ceph Storage cluster in the Installation Guide.
Procedure
To change the variable defaults in director, use the
CephAnsibleExtraConfig
parameter to pass the new values in heat environment files. For example, to set theceph-ansible
group variablejournal_size
to 40960, create an environment file with the followingjournal_size
definition:parameter_defaults: CephAnsibleExtraConfig: journal_size: 40960
ImportantChange
ceph-ansible
group variables with the override parameters; do not edit group variables directly in the/usr/share/ceph-ansible
directory on the undercloud.
5.2. Ceph containers for Red Hat OpenStack Platform with Ceph Storage
To configure Red Hat OpenStack Platform (RHOSP) to use Red Hat Ceph Storage with NFS Ganesha, you must have a Ceph container.
To be compatible with Red Hat Enterprise Linux 8, RHOSP 16 requires Red Hat Ceph Storage 4 or 5 (Ceph package 14.x or Ceph package 16.x). The Ceph Storage 4 and 5 containers are hosted at registry.redhat.io
, a registry that requires authentication. For more information, see Container image preparation parameters.
5.3. Mapping the Ceph Storage node disk layout
When you deploy containerized Ceph Storage, you must map the disk layout and specify dedicated block devices for the Ceph OSD service. You can perform this mapping in the environment file that you created earlier to define your custom Ceph parameters: /home/stack/templates/ceph-config.yaml
.
Use the CephAnsibleDisksConfig
resource in parameter_defaults
to map your disk layout. This resource uses the following variables:
Variable | Required? | Default value (if unset) | Description |
---|---|---|---|
osd_scenario | Yes | lvm
NOTE: The default value is |
The |
devices | Yes | NONE. Variable must be set. | A list of block devices that you want to use for OSDs on the node. |
dedicated_devices |
Yes (only if | devices |
A list of block devices that maps each entry in the |
dmcrypt | No | false |
Sets whether data stored on OSDs is encrypted ( |
osd_objectstore | No | bluestore
NOTE: The default value is | Sets the storage back end used by Ceph.
NOTE: Although the value defaults to |
5.3.1. Using BlueStore
Procedure
To specify the block devices that you want to use as Ceph OSDs, use a variation of the following snippet:
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/nvme0n1 osd_scenario: lvm osd_objectstore: bluestore
Because
/dev/nvme0n1
is in a higher performing device class, the exampleparameter_defaults
produces three OSDs that run on/dev/sdb
,/dev/sdc
, and/dev/sdd
. The three OSDs use/dev/nvme0n1
as theblock.db
and BlueStore WAL device. Theceph-volume
tool does this by using thebatch
subcommand. The same configuration is duplicated for each Ceph Storage node and assumes uniform hardware. If theblock.db
and BlueStore WAL data reside on the same disks as the OSDs, then change the parameter defaults in the following way:parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd osd_scenario: lvm osd_objectstore: bluestore
5.3.2. Referring to devices with persistent names
Procedure
In some nodes, disk paths, such as
/dev/sdb
and/dev/sdc
, might not point to the same block device during reboots. If this is the case with your Ceph Storage nodes, specify each disk with the/dev/disk/by-path/
symlink to ensure consistent block device mapping throughout your deployments:parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0 - /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:11:0 dedicated_devices: - /dev/nvme0n1 - /dev/nvme0n1
Optional: Because you must set the list of OSD devices before overcloud deployment, it might not be possible to identify and set the PCI path of disk devices. In this case, gather the
/dev/disk/by-path/symlink
data for block devices during introspection.In the following example, run the first command to download the introspection data from the undercloud Object Storage service (swift) for the server
b08-h03-r620-hci
and save the data in a file calledb08-h03-r620-hci.json
. Run the second command to grep for “by-path”. The output of this command contains the unique/dev/disk/by-path
values that you can use to identify disks.(undercloud) [stack@b08-h02-r620 ironic]$ openstack baremetal introspection data save b08-h03-r620-hci | jq . > b08-h03-r620-hci.json (undercloud) [stack@b08-h02-r620 ironic]$ grep by-path b08-h03-r620-hci.json "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:0:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:1:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:3:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:4:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:5:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:6:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:7:0", "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:0:0",
For more information about naming conventions for storage devices, see Overview of persistent naming attributes in the Managing storage devices guide.
5.3.3. Configuring OSDs in advanced scenarios
In an environment file, you list the block devices that you want to use for OSDs in the devices
variable of the CephAnsibleDisksConfig
resource.
When you use the devices
variable without any other device configuration parameter, ceph-volume lvm batch
automatically optimizes OSD configuration by evenly sharing the higher performance device as a block.db
for the slower devices.
You can use the following procedures to configure devices
to avoid running in ceph-volume lvm batch
mode.
5.3.3.1. Using a block.db to improve performance
Using a block.db
can improve the performance of your Ceph Storage cluster by increasing throughput and improving response times. A block.db
is a database that consists of data segments and BlueStore write-ahead logs (WAL).
Procedure
Add the following content to an environment file:
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sda - /dev/sdb - /dev/nvme0n1 - /dev/sdc - /dev/sdd - /dev/nvme0n2 osd_scenario: lvm osd_objectstore: bluestore
This configures four OSDs:
sda
,sdb
,sdc
, andsdd
. Each pair has its own database:nvem0n1
andnvme0n2
.NoteThe order of the devices in the
devices
list is significant. List the drives followed by the block.db and BlueStore WAL (DB-WAL) device. In the example,nvme0n1
is the DB-WAL forsda
andsdb
andnvme0n2
is the DB-WAL forsdc
andsdd
. For more information, see Using BlueStore.-
Include the environment file that contains your new content in the deployment command with the
-e
option when you deploy the overcloud.
5.3.3.2. Using dedicated write-ahead log (WAL) devices
You can specify a dedicated write-ahead log (WAL) device. Using devices
, dedicated_devices
, and bluestore_wal_devices
together means that you can isolate all components of an OSD on to separate devices, which can increase performance.
In the following example procedure, another additional dictionary, bluestore_wal_devices
, isolates the write-ahead log on NVMe devices nvme0n1
and nvme0n2
.
Procedure
Add the following content to an environment file:
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sda - /dev/sdb dedicated_devices: - /dev/sdx - /dev/sdy bluestore_wal_devices: - /dev/nvme0n1 - /dev/nvme0n2
-
Include the environment file that contains your new content in the deployment command with the
-e
option when you deploy the overcloud.
5.3.3.3. Using pre-created LVMs for increased control
In the previous advanced scenarios, ceph-volume
uses different types of device lists to create logical volumes for OSDs. You can also create the logical volumes before ceph-volume
runs and then pass ceph-volume
an lvm_volumes
list of those logical volumes. Although this requires that you create the logical volumes in advance, it means that you have more precise control. Because director is also responsible for hardware provisioning, you must create these LVMs in advance by using a first-boot script.
Procedure
Create an environment file,
/home/stack/templates/firstboot.yaml
, that registers your heat template as theOS::TripleO::NodeUserData
resource type and contains the following content:resource_registry: OS::TripleO::NodeUserData: /home/stack/templates/ceph-lvm.yaml
Create an environment file,
/home/stack/templates/ceph-lvm.yaml
. Add a list similar to the following example, which includes three physical volumes. If your devices list is longer, expand the example according to your requirements.heat_template_version: 2014-10-16 description: > Extra hostname configuration resources: userdata: type: OS::Heat::MultipartMime properties: parts: - config: {get_resource: ceph_lvm_config} ceph_lvm_config: type: OS::Heat::SoftwareConfig properties: config: | #!/bin/bash -x pvcreate /dev/sda vgcreate ceph_vg_hdd /dev/sda pvcreate /dev/sdb vgcreate ceph_vg_ssd /dev/sdb pvcreate /dev/nvme0n1 vgcreate ceph_vg_nvme /dev/nvme0n1 lvcreate -n ceph_lv_wal1 -L 50G ceph_vg_nvme lvcreate -n ceph_lv_db1 -L 500G ceph_vg_ssd lvcreate -n ceph_lv_data1 -L 5T ceph_vg_hdd lvs outputs: OS::stack_id: value: {get_resource: userdata}
Use the
lvm_volumes
parameter instead of the devices list in the following way. This assumes that the volume groups and logical volumes are already created. A typical use case in this scenario is that the WAL and DB LVs are on SSDs and the data LV is on HDDs:parameter_defaults: CephAnsibleDisksConfig: osd_objectstore: bluestore osd_scenario: lvm lvm_volumes: - data: ceph_lv_data1 data_vg: ceph_vg_hdd db: ceph_lv_db1 db_vg: ceph_vg_ssd wal: ceph_lv_wal1 wal_vg: ceph_vg_nvme
Include the environment files that contain your new content in the deployment command with the
-e
option when you deploy the overcloud.- Note
- Specifying a separate WAL device is necessary only if that WAL device resides on hardware that performs better than the DB device. Usually creating a separate DB device is sufficient and the same partition is then used for the WAL function.
5.4. Assigning custom attributes to different Ceph pools
Use the CephPools
parameter to apply different attributes to each Ceph Storage pool or create a new custom pool.
Procedure
Replace
POOL
with the name of the pool that you want to configure:parameter_defaults: CephPools: - name: POOL
Configure placement groups by doing one of the following:
To manually override the default settings, set
pg_num
to the number of placement groups:parameter_defaults: CephPools: - name: POOL pg_num: 128 application: rbd
Alternatively, to automatically scale placement groups, set
pg_autoscale_mode
toTrue
and settarget_size_ratio
to a percentage relative to your expected Ceph Storage requirements:parameter_defaults: CephPools: - name: POOL pg_autoscale_mode: True target_size_ratio: PERCENTAGE application: rbd
Replace
PERCENTAGE
with a decimal. For example, 0.5 equals 50 percent. The total percentage must equal 1.0 or 100 percent.The following values are for example only:
paramter_defaults: CephPools: - {"name": backups, "target_size_ratio": 0.1, "pg_autoscale_mode": True, "application": rbd} - {"name": volumes, "target_size_ratio": 0.5, "pg_autoscale_mode": True, "application": rbd} - {"name": vms, "target_size_ratio": 0.2, "pg_autoscale_mode": True, "application": rbd} - {"name": images, "target_size_ratio": 0.2, "pg_autoscale_mode": True, "application": rbd}
For more information, see The placement group autoscaler in the Red Hat Ceph Storage Installation Guide.
Specify the application type.
The application type for Compute, Block Storage, and Image Storage is`rbd`. However, depending on what you use the pool for, you can specify a different application type.
For example, the application type for the gnocchi metrics pool is
openstack_gnocchi
. For more information, see Enable Application in the Storage Strategies Guide .NoteIf you do not use the
CephPools
parameter, director sets the appropriate application type automatically, but only for the default pool list.Optional: Add a pool called
custompool
to create a custom pool, and set the parameters specific to the needs of your environment:parameter_defaults: CephPools: - name: custompool pg_num: 128 application: rbd
This creates a new custom pool in addition to the default pools.
For typical pool configurations of common Ceph use cases, see the Ceph Placement Groups (PGs) per Pool Calculator. This calculator is normally used to generate the commands for manually configuring your Ceph pools. In this deployment, the director configures the pools based on your specifications.
Red Hat Ceph Storage 3 (Luminous) introduced a hard limit on the maximum number of PGs an OSD can have, which is 200 by default. Do not override this parameter beyond 200. If there is a problem because the Ceph PG number exceeds the maximum, adjust the pg_num
per pool to address the problem, not the mon_max_pg_per_osd
.
5.5. Overriding parameters for dissimilar Ceph Storage nodes
All nodes with a role that host Ceph OSDs, such as CephStorage
or ComputeHCI
, use the global devices
and dedicated_devices
lists created in Section 5.3, “Mapping the Ceph Storage node disk layout”. These lists assume all of these servers have the same hardware. If there are servers with hardware this is not the same, you must update director with the details of the different devices
and dedicated_devices
lists using node-specific disk configuration.
Roles that host Ceph OSDs include the OS::TripleO::Services::CephOSD service in the roles_data.yaml
file.
Ceph Storage nodes that do not have the same hardware as other nodes can cause performance issues. The more variance there is between a standard node and a node that you configure with node-specific overrides in your Red Hat OpenStack Platform (RHOSP) environment, the larger the possible performance penalty.
5.5.1. Node-specific disk configuration
Director must be configured for services with that do not have the same hardware. This is called node-specific disk configuration.
You can create your node-specific disk configuration by using one of the following methods:
- Automatic: You can generate a JSON heat environment file to automatically create the node-specific disk configuration.
- Manual: You can alter the node disk layout to create the node-specific disk configuration.
5.5.1.1. Generating a JSON heat environment file for Ceph devices
You can use the /usr/share/openstack-tripleo-heat-templates/tools/make_ceph_disk_list.py
script to create a valid JSON heat environment file automatically from the introspection data of the Bare Metal Provisioning service (ironic). Use this JSON file to pass a node-specific disk configuration to director.
Procedure
Export the introspection data from the Bare Metal Provisioning service for the Ceph nodes that you want to deploy:
openstack baremetal introspection data save oc0-ceph-0 > ceph0.json openstack baremetal introspection data save oc0-ceph-1 > ceph1.json ...
Copy the utility to the home directory of the
stack
user on the undercloud and use it to generate anode_data_lookup.json
file../make_ceph_disk_list.py -i ceph*.json -o node_data_lookup.json -k by_path
Pass the introspection data file from the
openstack baremetal introspection data save
command for all nodes that host Ceph OSDs to the utility because you can only defineNodeDataLookup
once during a deployment. The-i
option can take an expression like*.json
or a list of files as input.Use the
-k
option to define the key of the Bare Metal Provisioning disk data structure that you want to use to identify your OSD disk. Do not usename
because it produces a file of devices like/dev/sdd
, which might not always point to the same device during a reboot. Instead, useby_path
. This is the default if you do not specify-k
.The Bare Metal Provisioning service reserves one of the available disks on the system as the root disk. The utility always excludes the root disk from the list of generated devices.
-
Optional: You can use
./make_ceph_disk_list.py –help
to see other available options. Include the
node_data_lookup.json
file with any other environment files that are relevant to your environment when you deploy the overcloud:$ openstack overcloud deploy \ --templates \ … -e <existing_overcloud_environment_files> \ -e node_data_lookup.json \ …
5.5.1.2. Altering the disk layout in Ceph Storage nodes
Non-homogeneous Ceph Storage nodes can cause performance issues. The more variance there is between a standard node and a node that you configure with node-specific overrides in your Red Hat OpenStack Platform (RHOSP) environment, the larger the possible performance penalty.
To pass a node-specific disk configuration to director, you must pass a heat environment file, such as node-spec-overrides.yaml
, to the openstack overcloud deploy
command and the file content must identify each server by a machine-unique UUID and a list of local variables to override the global variables.
You can extract the machine-unique UUID for each individual server or from the Bare Metal Provisioning service (ironic) database.
- Note
-
In the following procedure you create a valid YAML environment file that contains embedded valid JSON. You can also generate a full JSON file with
make_ceph_disk_list.py
and pass it to the deployment command as if it were YAML. For more information, see Generating a JSON heat environment file for Ceph devices.
Procedure
To locate the UUID for an individual server, log in to the server and enter the following command:
$ dmidecode -s system-uuid
To extract the UUID from the the Bare Metal Provisioning service database, enter the following command on the undercloud:
$ openstack baremetal introspection data save NODE-ID | jq .extra.system.product.uuid
WarningIf the
undercloud.conf
does not haveinspection_extras = true
before undercloud installation or upgrade and introspection, then the machine-unique UUID is not in the Bare Metal Provisioning service database.ImportantThe machine-unique UUID is not the Bare Metal Provisioning service UUID.
A valid
node-spec-overrides.yaml
file might look like the following:parameter_defaults: NodeDataLookup: {"32E87B4C-C4A7-418E-865B-191684A6883B": {"devices": ["/dev/sdc"]}}
All lines after the first two lines must be valid JSON. Use the
jq
command to verify that the JSON is valid.-
Remove the first two lines (
parameter_defaults:
andNodeDataLookup:
) from the file temporarily. -
Enter
cat node-spec-overrides.yaml | jq .
-
Remove the first two lines (
As the
node-spec-overrides.yaml
file grows, you can also use thejq
command to ensure that the embedded JSON is valid. For example, because thedevices
anddedicated_devices
list must be the same length, use the following command to verify that they are the same length before you start the deployment. In the following example, thenode-spec-c05-h17-h21-h25-6048r.yaml
has three servers in rack c05 in which slots h17, h21, and h25 are missing disks.(undercloud) [stack@b08-h02-r620 tht]$ cat node-spec-c05-h17-h21-h25-6048r.yaml | jq '.[] | .devices | length' 33 30 33 (undercloud) [stack@b08-h02-r620 tht]$ cat node-spec-c05-h17-h21-h25-6048r.yaml | jq '.[] | .dedicated_devices | length' 33 30 33 (undercloud) [stack@b08-h02-r620 tht]$
After the JSON is validated, add back the two lines that make it a valid environment YAML file (
parameter_defaults:
andNodeDataLookup:
) and include it with-e
in the deployment command.In the following example, the updated heat environment file uses
NodeDataLookup
for Ceph deployment. All of the servers had a devices list with 35 disks except one of them had a disk missing. This environment file overrides the default devices list for only that single node and supplies the node with the list of 34 disks that it must use instead of the global list.parameter_defaults: # c05-h01-6048r is missing scsi-0:2:35:0 (00000000-0000-0000-0000-0CC47A6EFD0C) NodeDataLookup: { "00000000-0000-0000-0000-0CC47A6EFD0C": { "devices": [ "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:32:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:2:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:3:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:4:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:5:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:6:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:33:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:7:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:8:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:34:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:9:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:10:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:11:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:12:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:13:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:14:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:15:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:16:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:17:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:18:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:19:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:20:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:21:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:22:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:23:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:24:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:25:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:26:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:27:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:28:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:29:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:30:0", "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:31:0" ], "dedicated_devices": [ "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:81:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1", "/dev/disk/by-path/pci-0000:84:00.0-nvme-1" ] } }
5.5.2. Altering the BlueStore block.db size
The BlueStore block.db
is a database of data segments and BlueStore write-ahead logs (WAL). There are two methods for altering the database size. Select one of these methods to alter the size.
5.5.2.1. Altering the BlueStore block.db
size when you use ceph-volume
Use the following procedure to override the block.db
size when you use ceph-volume
. ceph-volume
is used when osd_scenario: lvm
. ceph-volume
automatically sets the block.db
size. However, you can override the block.db
size for advanced scenarios.
The following example uses a ceph-ansible
host variable, not a Ceph configuration file override, so that the block_db_size
that is used is passed to the ceph-volume
call.
Procedure
Create a JSON environment file with content similar to the following but replace the values according to your requirements:
{ "parameter_defaults": { "NodeDataLookup": { "32e87b4c-c4a7-41be-865b-191684a6883b": { "block_db_size": 3221225472 }, "ea6a84d6-cf89-4fe2-b7bd-869b3fe4dd6b": { "block_db_size": 3221225472 } } } }
Include the JSON file with any other environment files that are relevant to your environment when you deploy the overcloud:
$ openstack overcloud deploy \ --templates \ … -e <existing_overcloud_environment_files> \ -e <json_environment_file> \ …
5.5.2.2. Altering the BlueStore block.db
size when you use ceph-disk
Use the following procedure to override the block.db
size when you use ceph-disk
. ceph-disk
is used when osd_scenario: non-collocated
or osd_scenario: collocated
.
The following example uses a Ceph configuration override for specific nodes to set the blustore_block_db_size
. This Ceph configuration option is ignored when you use ceph-volume
, however ceph-disk
uses this configuration option.
Procedure
Create a JSON environment file with content similar to the following but replace the values according to your requirements:
{ "parameter_defaults": { "NodeDataLookup": { "32e87b4c-c4a7-41be-865b-191684a6883b": { "ceph_conf_overrides": { "osd": { "bluestore_block_db_size": 3221225472 } } }, "ea6a84d6-cf89-4fe2-b7bd-869b3fe4dd6b": { "ceph_conf_overrides": { "osd": { "bluestore_block_db_size": 3221225472 } } } } } }
Include the JSON file with any other environment files that are relevant to your environment when you deploy the overcloud:
$ openstack overcloud deploy \ --templates \ … -e <existing_overcloud_environment_files> \ -e <json_environment_file> \ …
5.6. Increasing the restart delay for large Ceph clusters
During deployment, Ceph services such as OSDs and Monitors, are restarted and the deployment does not continue until the service is running again. Ansible waits 15 seconds (the delay) and checks 5 times for the service to start (the retries). If the service does not restart, the deployment stops so the operator can intervene.
Depending on the size of the Ceph cluster, you may need to increase the retry or delay values. The exact names of these parameters and their defaults are as follows:
health_mon_check_retries: 5 health_mon_check_delay: 15 health_osd_check_retries: 5 health_osd_check_delay: 15
Procedure
Update the
CephAnsibleExtraConfig
parameter to change the default delay and retry values:parameter_defaults: CephAnsibleExtraConfig: health_osd_check_delay: 40 health_osd_check_retries: 30 health_mon_check_delay: 20 health_mon_check_retries: 10
This example makes the cluster check 30 times and wait 40 seconds between each check for the Ceph OSDs, and check 20 times and wait 10 seconds between each check for the Ceph MONs.
-
To incorporate the changes, pass the updated
yaml
file with-e
usingopenstack overcloud deploy
.
5.7. Overriding Ansible environment variables
The Red Hat OpenStack Platform Workflow service (mistral) uses Ansible to configure Ceph Storage, but you can customize the Ansible environment by using Ansible environment variables.
Procedure
To override an
ANSIBLE_*
environment variable, use theCephAnsibleEnvironmentVariables
heat template parameter. This example configuration increases the number of forks and SSH retries:parameter_defaults: CephAnsibleEnvironmentVariables: ANSIBLE_SSH_RETRIES: '6' DEFAULT_FORKS: '35'
For more information about Ansible environment variables, see Ansible Configuration Settings.
For more information about how to customize your Ceph Storage cluster, see Customizing the Ceph Storage cluster.
5.8. Enabling Ceph on-wire encryption
Starting with Red Hat Ceph Storage 4 and later, you can enable encryption for all Ceph traffic over the network with the introduction of the messenger version 2 protocol. The secure mode
setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.
This feature is intended for use with Red Hat OpenStack Platform (RHOSP) versions 16.1 and later. It is not supported on RHOSP version 13 deployments that use external Red Hat Ceph Storage version 4. For more information, see Ceph on-wire encryption in the Red Hat Ceph Storage Architecture Guide.
Procedure
To enable Ceph on-wire encryption in RHOSP, configure the following parameter in a new or an existing custom environment file:
parameter_defaults: CephMsgrSecureMode: true
After you update the environment file, redeploy the overcloud:
$ openstack overcloud deploy --templates -e <environment_file>
After you implement this change, director configures the Ceph Storage cluster with the following settings:
ms_cluster_mode: secure ms_service_mode: secure ms_client_mode: secure
For more information about Ceph on-wire encryption, see Ceph on-wire encryption in the Architecture Guide.