Chapter 5. Customizing the Ceph Storage cluster

Director deploys containerized Red Hat Ceph Storage using a default configuration. You can customize Ceph Storage by overriding the default settings.

Prerequistes

To deploy containerized Ceph Storage you must include the /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml file during overcloud deployment. This environment file defines the following resources:

Use these resources to override any defaults that the director sets for containerized Ceph Storage.

Procedure

  1. Enable the Red Hat Ceph Storage 4 Tools repository:

    $ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
  2. Install the ceph-ansible package on the undercloud:

    $ sudo dnf install ceph-ansible
  3. To customize your Ceph Storage cluster, define custom parameters in a new environment file, for example, /home/stack/templates/ceph-config.yaml. You can apply Ceph Storage cluster settings with the following syntax in the parameter_defaults section of your environment file:

    parameter_defaults:
      CephConfigOverrides:
        section:
          KEY:VALUE
    Note

    You can apply the CephConfigOverrides parameter to the [global] section of the ceph.conf file, as well as any other section, such as [osd], [mon], and [client]. If you specify a section, the key:value data goes into the specified section. If you do not specify a section, the data goes into the [global] section by default. For information about Ceph Storage configuration, customization, and supported parameters, see Red Hat Ceph Storage Configuration Guide.

  4. Replace KEY and VALUE with the Ceph cluster settings that you want to apply. For example, in the global section, max_open_files is the KEY and 131072 is the corresponding VALUE:

    parameter_defaults:
      CephConfigOverrides:
        global:
          max_open_files: 131072
        osd:
          osd_scrub_during_recovery: false

    This configuration results in the following settings defined in the configuration file of your Ceph cluster:

    [global]
    max_open_files = 131072
    [osd]
    osd_scrub_during_recovery = false

5.1. Setting ceph-ansible group variables

The ceph-ansible tool is a playbook used to install and manage Ceph Storage clusters.

The ceph-ansible tool has a group_vars directory that defines configuration options and the default settings for those options. Use the group_vars directory to set Ceph Storage parameters.

For information about the group_vars directory, see Installing a Red Hat Ceph Storage cluster in the Installation Guide.

To change the variable defaults in director, use the CephAnsibleExtraConfig parameter to pass the new values in heat environment files. For example, to set the ceph-ansible group variable journal_size to 40960, create an environment file with the following journal_size definition:

parameter_defaults:
  CephAnsibleExtraConfig:
    journal_size: 40960
Important

Change ceph-ansible group variables with the override parameters; do not edit group variables directly in the /usr/share/ceph-ansible directory on the undercloud.

5.2. Ceph containers for Red Hat OpenStack Platform with Ceph Storage

A Ceph container is required to configure OpenStack Platform to use Ceph, even with an external Ceph cluster. To be compatible with Red Hat Enterprise Linux 8, Red Hat OpenStack Platform (RHOSP) 15 requires Red Hat Ceph Storage 4. The Ceph Storage 4 container is hosted at registry.redhat.io, a registry that requires authentication.

You can use the heat environment parameter ContainerImageRegistryCredentials to authenticate at registry.redhat.io. For more information, see Container image preparation parameters.

5.3. Mapping the Ceph Storage node disk layout

When you deploy containerized Ceph Storage, you must map the disk layout and specify dedicated block devices for the Ceph OSD service. You can perform this mapping in the environment file that you created earlier to define your custom Ceph parameters: /home/stack/templates/ceph-config.yaml.

Use the CephAnsibleDisksConfig resource in parameter_defaults to map your disk layout. This resource uses the following variables:

VariableRequired?Default value (if unset)Description

osd_scenario

Yes

lvm

NOTE: The default value is lvm.

The lvm value allows ceph-ansible to use ceph-volume to to configure OSDs, block.db, and BlueStore WAL devices.

devices

Yes

NONE. Variable must be set.

A list of block devices that you want to use for OSDs on the node.

dedicated_devices

Yes (only if osd_scenario is non-collocated)

devices

A list of block devices that maps each entry in the devices parameter to a dedicated journaling block device. You can use this variable only when osd_scenario=non-collocated.

dmcrypt

No

false

Sets whether data stored on OSDs is encrypted (true) or unencrypted (false).

osd_objectstore

No

bluestore

NOTE: The default value is bluestore.

Sets the storage back end used by Ceph.

5.3.1. Using BlueStore

Procedure

  1. To specify the block devices that you want to use as Ceph OSDs, use a variation of the following snippet:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
          - /dev/nvme0n1
        osd_scenario: lvm
        osd_objectstore: bluestore
  2. Because /dev/nvme0n1 is in a higher performing device class, the example parameter defaults produce three OSDs that run on /dev/sdb, /dev/sdc, and /dev/sdd. The three OSDs use /dev/nvme0n1 as the block.db and BlueStore WAL device. The ceph-volume tool does this by using the batch subcommand. The same configuration is duplicated for each Ceph Storage node and assumes uniform hardware. If the block.db and BlueStore WAL data resides on the same disks as the OSDs, then change the parameter defaults in the following way:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
        osd_scenario: lvm
        osd_objectstore: bluestore

5.3.2. Referring to devices with persistent names

Procedure

  1. In some nodes, disk paths, such as /dev/sdb and /dev/sdc, might not point to the same block device during reboots. If this is the case with your CephStorage nodes, specify each disk with the /dev/disk/by-path/ symlink to ensure that the block device mapping is consistent throughout deployments:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
    
          - /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:10:0
          - /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:11:0
    
    
        dedicated_devices:
          - /dev/nvme0n1
          - /dev/nvme0n1
  2. Optional: Because you must set the list of OSD devices before overcloud deployment, it might not be possible to identify and set the PCI path of disk devices. In this case, gather the /dev/disk/by-path/symlink data for block devices during introspection.

    In the following example, run the first command to download the introspection data from the undercloud Object Storage service (swift) for the server b08-h03-r620-hci and save the data in a file called b08-h03-r620-hci.json. Run the second command to grep for “by-path”. The output of this command contains the unique /dev/disk/by-path values that you can use to identify disks.

    (undercloud) [stack@b08-h02-r620 ironic]$ openstack baremetal introspection data save b08-h03-r620-hci | jq . > b08-h03-r620-hci.json
    (undercloud) [stack@b08-h02-r620 ironic]$ grep by-path b08-h03-r620-hci.json
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:0:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:1:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:3:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:4:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:5:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:6:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:7:0",
            "by_path": "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:2:0:0",

For more information about naming conventions for storage devices, see Overview of persistent naming attributes in the Managing storage devices guide.

5.3.3. Configuring OSDs in advanced scenarios

The devices variable is where you list block devices that you want to use for OSDs on a node with the CephAnsibleDisksConfig resource in the parameter_defaults section of an environment file. When you use the devices variable without any other device configuration parameter, ceph-volume lvm batch automatically optimizes OSD configuration by evenly sharing the higher performance device as a block.db for the slower devices. However, you can use the following advanced scenarios, so that the devices option does not result in ceph-volume lvm batch mode running.

5.3.3.1. Specifying the block.db device

You can specify a block.db device to increase performance.

In the following example procedure, there are two OSDs. The sda and sdb devices each have their own data segments and write-ahead logs. The additional dictionary, dedicated_devices, isolates their databases, also known as block.db, on sdx and sdy.

Procedure

  1. Add the following content to an environment file:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sda
          - /dev/sdb
        dedicated_devices:
          - /dev/sdx
          - /dev/sdy
  2. Include the environment file that contains your new content in the deployment command with the -e option when you deploy the overcloud.

5.3.3.2. Using dedicated write-ahead log (WAL) devices

You can specify a dedicated write-ahead log (WAL) device. Using devices, dedicated_devices, and bluestore_wal_devices together means that you can isolate all components of an OSD on to separate devices, which can increase performance.

In the following example procedure, another additional dictionary, bluestore_wal_devices, isolates the write-ahead log on NVMe devices nvme0n1 and nvme0n2.

Procedure

  1. Add the following content to an environment file:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sda
          - /dev/sdb
        dedicated_devices:
          - /dev/sdx
          - /dev/sdy
        bluestore_wal_devices:
          - /dev/nvme0n1
          - /dev/nvme0n2
  2. Include the environment file that contains your new content in the deployment command with the -e option when you deploy the overcloud.

5.3.3.3. Using pre-created LVMs for increased control

In the previous advanced scenarios, ceph-volume uses different types of device lists to create logical volumes for OSDs. You can also create the logical volumes before ceph-volume runs and then pass ceph-volume an lvm_volumes list of those logical volumes. Although this requires that you create the logical volumes in advance, it means that you have more precise control. Because director is also responsible for hardware provisioning, you must create these LVMs in advance by using a first-boot script.

Procedure

  1. Create an environment file, /home/stack/templates/firstboot.yaml, that registers your heat template as the OS::TripleO::NodeUserData resource type and contains the following content:

    resource_registry:
      OS::TripleO::NodeUserData: /home/stack/templates/ceph-lvm.yaml
  2. Create an environment file, /home/stack/templates/ceph-lvm.yaml. Add a list similar to the following example, which includes three physical volumes. If your devices list is longer, expand the example according to your requirements.

    heat_template_version: 2014-10-16
    
    description: >
      Extra hostname configuration
    
    resources:
      userdata:
        type: OS::Heat::MultipartMime
        properties:
          parts:
          - config: {get_resource: ceph_lvm_config}
    
      ceph_lvm_config:
        type: OS::Heat::SoftwareConfig
        properties:
          config: |
            #!/bin/bash -x
            pvcreate /dev/sda
            vgcreate ceph_vg_hdd /dev/sda
            pvcreate /dev/sdb
            vgcreate ceph_vg_ssd /dev/sdb
            pvcreate /dev/nvme0n1
            vgcreate ceph_vg_nvme /dev/nvme0n1
            lvcreate -n ceph_lv_wal1 -L 50G ceph_vg_nvme
            lvcreate -n ceph_lv_db1 -L 500G ceph_vg_ssd
            lvcreate -n ceph_lv_data1 -L 5T ceph_vg_hdd
            lvs
    
    outputs:
      OS::stack_id:
        value: {get_resource: userdata}
  3. Use the lvm_volumes parameter instead of the devices list in the following way. This assumes that the volume groups and logical volumes are already created. A typical use case in this scenario is that the WAL and DB LVs are on SSDs and the data LV is on HDDs:

    parameter_defaults:
      CephAnsibleDisksConfig:
        osd_objectstore: bluestore
        osd_scenario: lvm
        lvm_volumes:
          - data: ceph_lv_data1
            data_vg: ceph_vg_hdd
            db: ceph_lv_db1
            db_vg: ceph_vg_ssd
            wal: ceph_lv_wal1
            wal_vg: ceph_vg_nvme
  4. Include the environment files that contain your new content in the deployment command with the -e option when you deploy the overcloud.

    Note
    Specifying a separate WAL device is necessary only if that WAL device resides on hardware that performs better than the DB device. Usually creating a separate DB device is sufficient and the same partition is then used for the WAL function.

5.4. Assigning custom attributes to different Ceph pools

By default, Ceph Storage pools created with director have the same number of placement groups (pg_num and pgp_num) and sizes. You can use either method in Chapter 5, Customizing the Ceph Storage cluster to override these settings globally. Doing so applies the same values to all pools.

Use the CephPools parameter to apply different attributes to each Ceph Storage pool or create a new custom pool.

Procedure

  1. Replace POOL with the name of the pool that you want to configure:

    parameter_defaults:
      CephPools:
        - name: POOL
  2. Configure placement groups by doing one of the following:

    • To manually override the default settings, set pg_num to the number of placement groups:

      parameter_defaults:
        CephPools:
          - name: POOL
            pg_num: 128
            application: rbd
    • Alternatively, to automatically scale placement groups, set pg_autoscale_mode to True and set target_size_ratio to a percentage relative to your expected Ceph Storage requirements:

      parameter_defaults:
        CephPools:
          - name: POOL
            pg_autoscale_mode: True
            target_size_ratio: PERCENTAGE
            application: rbd

      Replace PERCENTAGE with a decimal. For example, 0.5 equals 50 percent. The total percentage must equal 1.0 or 100 percent.

      The following values are for example only:

      paramter_defaults:
        CephPools:
          - {"name": backups, "target_size_ratio": 0.1, "pg_autoscale_mode": True, "application": rbd}
          - {"name": volumes, "target_size_ratio": 0.5, "pg_autoscale_mode": True, "application": rbd}
          - {"name": vms,     "target_size_ratio": 0.2, "pg_autoscale_mode": True, "application": rbd}
          - {"name": images,  "target_size_ratio": 0.2, "pg_autoscale_mode": True, "application": rbd}

      For more information, see The placement group autoscaler in the Red Hat Ceph Storage Installation Guide.

  3. Specify the application type.

    The application type for Compute, Block Storage, and Image Storage is`rbd`. However, depending on what you use the pool for, you can specify a different application type.

    For example, the application type for the gnocchi metrics pool is openstack_gnocchi. For more information, see Enable Application in the Storage Strategies Guide .

    Note

    If you do not use the CephPools parameter, director sets the appropriate application type automatically, but only for the default pool list.

  4. Optional: Add a pool called custompool to create a custom pool, and set the parameters specific to the needs of your environment:

    parameter_defaults:
      CephPools:
        - name: custompool
          pg_num: 128
          application: rbd
Tip

For typical pool configurations of common Ceph use cases, see the Ceph Placement Groups (PGs) per Pool Calculator. This calculator is normally used to generate the commands for manually configuring your Ceph pools. In this deployment, the director configures the pools based on your specifications.

Warning

Red Hat Ceph Storage 3 (Luminous) introduced a hard limit on the maximum number of PGs an OSD can have, which is 200 by default. Do not override this parameter beyond 200. If there is a problem because the Ceph PG number exceeds the maximum, adjust the pg_num per pool to address the problem, not the mon_max_pg_per_osd.

5.5. Mapping the disk layout to non-homogeneous Ceph Storage nodes

Important

Non-homogeneous Ceph Storage nodes can cause performance issues, such as the risk of unpredictable performance loss. Although you can configure non-homogeneous Ceph Storage nodes in your Red Hat OpenStack Platform environment, Red Hat does not recommend it.

By default, all nodes of a role that host Ceph OSDs (indicated by the OS::TripleO::Services::CephOSD service in roles_data.yaml), for example CephStorage or ComputeHCI nodes, use the global devices and dedicated_devices lists set in Section 5.3, “Mapping the Ceph Storage node disk layout”. This assumes that all of these servers have homogeneous hardware. If a subset of these servers do not have homogeneous hardware, then director needs to be aware that each of these servers has different devices and dedicated_devices lists. This is known as a node-specific disk configuration.

To pass a node-specific disk configuration to director, you must pass a heat environment file, such as node-spec-overrides.yaml, to the openstack overcloud deploy command and the file content must identify each server by a machine unique UUID and a list of local variables to override the global variables.

You can extract the machine unique UUID for each individual server or from the Ironic database.

Procedure

  1. To locate the UUID for an individual server, log in to the server and enter the following command:

    dmidecode -s system-uuid
  2. To extract the UUID from the Ironic database, enter the following command on the undercloud.

    openstack baremetal introspection data save NODE-ID | jq .extra.system.product.uuid
    Warning

    If the undercloud.conf does not have inspection_extras = true before undercloud installation or upgrade and introspection, then the machine unique UUID is not in the Ironic database.

    Important

    The machine unique UUID is not the Ironic UUID.

    A valid node-spec-overrides.yaml file might look like the following:

    parameter_defaults:
      NodeDataLookup: {"32E87B4C-C4A7-418E-865B-191684A6883B": {"devices": ["/dev/sdc"]}}
  3. All lines after the first two lines must be valid JSON. You can verify that the JSON is valid by using the jq command:

    1. Remove the first two lines (parameter_defaults: and NodeDataLookup:) from the file temporarily.
    2. Run cat node-spec-overrides.yaml | jq .
  4. As the node-spec-overrides.yaml file grows, you can also use jq to ensure that the embedded JSON is valid. For example, because the devices and dedicated_devices list must be the same length, use the following to verify that they are the same length before you start the deployment.

    (undercloud) [stack@b08-h02-r620 tht]$ cat node-spec-c05-h17-h21-h25-6048r.yaml | jq '.[] | .devices | length'
    33
    30
    33
    (undercloud) [stack@b08-h02-r620 tht]$ cat node-spec-c05-h17-h21-h25-6048r.yaml | jq '.[] | .dedicated_devices | length'
    33
    30
    33
    (undercloud) [stack@b08-h02-r620 tht]$

    In the above example, the node-spec-c05-h17-h21-h25-6048r.yaml has three servers in rack c05 in which slots h17, h21, and h25 are missing disks. A more complicated example is included at the end of this section.

  5. After the JSON has been validated add back the two lines which makes it a valid environment YAML file (parameter_defaults: and NodeDataLookup:) and include it with -e in the deployment. In the example below, the updated heat environment file uses NodeDataLookup for Ceph deployment. All of the servers had a devices list with 35 disks except one of them had a disk missing. This environment file overrides the default devices list for only that single node and gives it the list of 34 disks it must use instead of the global list.

    parameter_defaults:
      # c05-h01-6048r is missing scsi-0:2:35:0 (00000000-0000-0000-0000-0CC47A6EFD0C)
      NodeDataLookup: {
        "00000000-0000-0000-0000-0CC47A6EFD0C": {
          "devices": [
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:32:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:2:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:3:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:4:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:5:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:6:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:33:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:7:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:8:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:34:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:9:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:10:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:11:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:12:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:13:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:14:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:15:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:16:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:17:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:18:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:19:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:20:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:21:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:22:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:23:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:24:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:25:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:26:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:27:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:28:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:29:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:30:0",
        "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:31:0"
            ],
          "dedicated_devices": [
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:81:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1",
        "/dev/disk/by-path/pci-0000:84:00.0-nvme-1"
            ]
          }
        }

5.6. Increasing the restart delay for large Ceph clusters

During deployment, Ceph services such as OSDs and Monitors, are restarted and the deployment does not continue until the service is running again. Ansible waits 15 seconds (the delay) and checks 5 times for the service to start (the retries). If the service does not restart, the deployment stops so the operator can intervene.

Depending on the size of the Ceph cluster, you may need to increase the retry or delay values. The exact names of these parameters and their defaults are as follows:

 health_mon_check_retries: 5
 health_mon_check_delay: 15
 health_osd_check_retries: 5
 health_osd_check_delay: 15

Procedure

  1. Update the CephAnsibleExtraConfig parameter to change the default delay and retry values:

    parameter_defaults:
      CephAnsibleExtraConfig:
        health_osd_check_delay: 40
        health_osd_check_retries: 30
        health_mon_check_delay: 20
        health_mon_check_retries: 10

    This example makes the cluster check 30 times and wait 40 seconds between each check for the Ceph OSDs, and check 20 times and wait 10 seconds between each check for the Ceph MONs.

  2. To incorporate the changes, pass the updated yaml file with -e using openstack overcloud deploy.

5.7. Overriding Ansible environment variables

The Red Hat OpenStack Platform Workflow service (mistral) uses Ansible to configure Ceph Storage, but you can customize the Ansible environment by using Ansible environment variables.

Procedure

To override an ANSIBLE_* environment variable, use the CephAnsibleEnvironmentVariables heat template parameter.

This example configuration increases the number of forks and SSH retries:

parameter_defaults:
  CephAnsibleEnvironmentVariables:
    ANSIBLE_SSH_RETRIES: '6'
    DEFAULT_FORKS: '35'

For more information about Ansible environment variables, see Ansible Configuration Settings.

For more information about how to customize your Ceph Storage cluster, see Customizing the Ceph Storage cluster.

5.8. Enabling Ceph on-wire encryption

Starting with Red Hat Ceph Storage 4 and later, you can enable encryption for all Ceph traffic over the network with the introduction of the messenger version 2 protocol. The secure mode setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.

This feature is available in this release as a Technology Preview, and therefore is not fully supported by Red Hat. It should only be used for testing, and should not be deployed in a production environment. For more information about Technology Preview features, see Scope of Coverage Details.

Note

This feature, although it is currently a Technology Preview, is intended for use with Red Hat OpenStack Platform (RHOSP) versions 16.1 and later. It is not supported on RHOSP version 13 deployments that use external Red Hat Ceph Storage version 4. For more information, see Ceph on-wire encryption in the Red Hat Ceph Storage Architecture Guide.

To enable Ceph on-wire encryption on RHOSP, set the following parameters in your environment override file:

parameter_defaults:
  CephConfigOverrides:
    global:
      ms_cluster_mode: secure
      ms_service_mode: secure
      ms_client_mode: secure

For more information about Ceph on-wire encryption, see Ceph on-wire encryption in the Architecture Guide.