Chapter 5. Installing Red Hat Ceph Storage using Ansible

Procedure

1. Log in as the root user account on the Ansible administration node.

2. For all deployments, bare-metal or in containers, install the ceph-ansible package:

   Red Hat Enterprise Linux 7

   [root@admin ~]# yum install ceph-ansible

   Red Hat Enterprise Linux 8

   [root@admin ~]# dnf install ceph-ansible

3. Navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

4. Create new yml files:

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml

   1. Bare-metal deployments:

      [root@admin ceph-ansible]# cp site.yml.sample site.yml

   2. Container deployments:

      [root@admin ceph-ansible]# cp site-container.yml.sample site-container.yml

5. Edit the new files.

   1. Open for editing the group_vars/all.yml file.

      Important

      Using a custom storage cluster name is not supported. Do not set the cluster parameter to any value other than ceph. Using a custom storage cluster name is only supported with Ceph clients, such as: librados, the Ceph Object Gateway, and RADOS block device mirroring.

      Warning

      By default, Ansible attempts to restart an installed, but masked firewalld service, which can cause the Red Hat Ceph Storage deployment to fail. To work around this issue, set the configure_firewall option to false in the all.yml file. If you are running the firewalld service, then there is no requirement to use the configure_firewall option in the all.yml file.

      Note

      Having the ceph_rhcs_version option set to 4 will pull in the latest version of Red Hat Ceph Storage 4.

      Note

      Red Hat recommends leaving the dashboard_enabled option set to True in the group_vars/all.yml file, and not changing it to False. If you want to disable the dashboard, see Disabling the Ceph Dashboard.

      Note

      Dashboard related components are containerized. Therefore, for Bare-metal or Container deployment, ceph_docker_registry_username and ceph_docker_registry_password parameters have to be included so that ceph-ansible can fetch container images required for the dashboard.

      Note

      If you do not have a Red Hat Registry Service Account, create one using the Registry Service Account webpage. See the Red Hat Container Registry Authentication Knowledgebase article for details on how to create and manage tokens.

      Note

      In addition to using a Service Account for the ceph_docker_registry_username and ceph_docker_registry_password parameters, you can also use your Customer Portal credentials, but to ensure security, encrypt the ceph_docker_registry_password parameter. For more information, see Encrypting Ansible password variables with ansible-vault.

      1. Bare-metal example of the all.yml file for CDN installation:

         fetch_directory: ~/ceph-ansible-keys
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

         Important

         Starting with Red Hat Ceph Storage 4.1, you must uncomment or set dashboard_admin_password and grafana_admin_password in /usr/share/ceph-ansible/group_vars/all.yml. Set secure passwords for each. Also set custom user names for dashboard_admin_user and grafana_admin_user.

         Note

         For Red Hat Ceph Storage 4.2, if you have used local registry for installation, use 4.6 for Prometheus image tags.

      2. Bare-metal example of the all.yml file for ISO installation:

         fetch_directory: ~/ceph-ansible-keys
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: iso
         ceph_rhcs_iso_path: /home/rhceph-4-rhel-8-x86_64.iso
         ceph_rhcs_version: 4
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

      3. Containers example of the all.yml file:

         fetch_directory: ~/ceph-ansible-keys
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_image: rhceph/rhceph-4-rhel8
         ceph_docker_image_tag: latest
         containerized_deployment: true
         ceph_docker_registry: registry.redhat.io
         ceph_docker_registry_auth: true
         ceph_docker_registry_username: SERVICE_ACCOUNT_USER_NAME
         ceph_docker_registry_password: TOKEN
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: registry.redhat.io/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: registry.redhat.io/openshift4/ose-prometheus:v4.6
         alertmanager_container_image: registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6

         1

         This is the interface on the public network.

         Important

         Look up the latest container images tags on the Red Hat Ecosystem Catalog to install the latest container images with all the latest patches applied.

      4. Containers example of the all.yml file, when the Red Hat Ceph Storage nodes do NOT have access to the Internet during deployment:

         fetch_directory: ~/ceph-ansible-keys
         monitor_interface: eth0 1
         public_network: 192.168.0.0/24
         ceph_docker_image: rhceph/rhceph-4-rhel8
         ceph_docker_image_tag: latest
         containerized_deployment: true
         ceph_docker_registry: LOCAL_NODE_FQDN:5000
         ceph_docker_registry_auth: false
         ceph_origin: repository
         ceph_repository: rhcs
         ceph_repository_type: cdn
         ceph_rhcs_version: 4
         dashboard_admin_user:
         dashboard_admin_password:
         node_exporter_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-node-exporter:v4.6
         grafana_admin_user:
         grafana_admin_password:
         grafana_container_image: LOCAL_NODE_FQDN:5000/rhceph/rhceph-4-dashboard-rhel8
         prometheus_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus:4.6
         alertmanager_container_image: LOCAL_NODE_FQDN:5000/openshift4/ose-prometheus-alertmanager:4.6

         1

         This is the interface on the public network.

         Replace

         • LOCAL_NODE_FQDN with your local host FQDN.

      5. From Red Hat Ceph Storage 4.2, dashboard_protocol is set to https and Ansible generates the dashboard and grafana keys and certificates. For custom certificates, in the all.yml file, update the path at Ansible installer host for dashboard_crt, dashboard_key, grafana_crt, and grafana_key for bare-metal or container deployment.

         Syntax

         dashboard_protocol: https
         dashboard_port: 8443
         dashboard_crt: 'DASHBOARD_CERTIFICATE_PATH'
         dashboard_key: 'DASHBOARD_KEY_PATH'
         dashboard_tls_external: false
         dashboard_grafana_api_no_ssl_verify: "{{ True if dashboard_protocol == 'https' and not grafana_crt and not grafana_key else False }}"
         grafana_crt: 'GRAFANA_CERTIFICATE_PATH'
         grafana_key: 'GRAFANA_KEY_PATH'

   2. To install Red Hat Ceph Storage using a container registry reachable with a http or https proxy, set the ceph_docker_http_proxy or ceph_docker_https_proxy variables in the group_vars/all.yml file.

      Example

      ceph_docker_http_proxy: http://192.168.42.100:8080
      ceph_docker_https_proxy: https://192.168.42.100:8080

      If you need to exclude some host for the proxy configuration, use the ceph_docker_no_proxy variable in the group_vars/all.yml file.

      Example

      ceph_docker_no_proxy: "localhost,127.0.0.1"

   3. In addition to editing the all.yml file for proxy installation of Red Hat Ceph Storage, edit the /etc/environment file:

      Example

      HTTP_PROXY: http://192.168.42.100:8080
      HTTPS_PROXY: https://192.168.42.100:8080
      NO_PROXY: "localhost,127.0.0.1"

      This triggers the podman to start the containerized services such as prometheus, grafana-server, alertmanager, and node-exporter, and download the required images.

   4. For all deployments, bare-metal or in containers, edit the group_vars/osds.yml file.

      Important

      Do not install an OSD on the device the operating system is installed on. Sharing the same device between the operating system and OSDs causes performance issues.

      Ceph-ansible uses the ceph-volume tool to prepare storage devices for Ceph usage. You can configure osds.yml to use your storage devices in different ways to optimize performance for your particular workload.

      Important

      All the examples below use the BlueStore object store, which is the format Ceph uses to store data on devices. Previously, Ceph used FileStore as the object store. This format is deprecated for new Red Hat Ceph Storage 4.0 installs because BlueStore offers more features and improved performance. It is still possible to use FileStore, but using it requires a Red Hat support exception. For more information on BlueStore, see Ceph BlueStore in the Red Hat Ceph Storage Architecture Guide.

      1. Auto discovery

         osd_auto_discovery: true

         The above example uses all empty storage devices on the system to create the OSDs, so you do not have to specify them explicitly. The ceph-volume tool checks for empty devices, so devices which are not empty will not be used.

         Note

         If you later decide to remove the cluster using purge-docker-cluster.yml or purge-cluster.yml, you must comment out osd_auto_discovery and declare the OSD devices in the osds.yml file. For more information, see Purging storage clusters deployed by Ansible.

      2. Simple configuration

         First Scenario

         devices:
           - /dev/sda
           - /dev/sdb

         or

         Second Scenario

         devices:
           - /dev/sda
           - /dev/sdb
           - /dev/nvme0n1
           - /dev/sdc
           - /dev/sdd
           - /dev/nvme1n1

         or

         Third Scenario

         lvm_volumes:
            - data: /dev/sdb
            - data: /dev/sdc

         or

         Fourth Scenario

         lvm_volumes:
             - data: /dev/sdb
             - data:/dev/nvme0n1

         When using the devices option alone, ceph-volume lvm batch mode automatically optimizes OSD configuration.

         In the first scenario, if the devices are traditional hard drives or SSDs, then one OSD per device is created.

         In the second scenario, when there is a mix of traditional hard drives and SSDs, the data is placed on the traditional hard drives (sda, sdb) and the BlueStore database is created as large as possible on the SSD (nvme0n1). Similarly, the data is placed on the traditional hard drives (sdc, sdd), and the BlueStore database is created on the SSD nvme1n1 irrespective of the order of devices mentioned.

         Note

         By default ceph-ansible does not override the default values of bluestore_block_db_size and bluestore_block_wal_size. You can set bluestore_block_db_size using ceph_conf_overrides in the group_vars/all.yml file. The value of bluestore_block_db_size should be greater than 2 GB.

         In the third scenario, data is placed on the traditional hard drives (sdb, sdc), and the BlueStore database is collocated on the same devices.

         In the fourth scenario, data is placed on the traditional hard drive (sdb) and on the SSD (nvme1n1), and the BlueStore database is collocated on the same devices. This is different from using the devices directive, where the BlueStore database is placed on the SSD.

         Important

         The ceph-volume lvm batch mode command creates the optimized OSD configuration by placing data on the traditional hard drives and the BlueStore database on the SSD. If you want to specify the logical volumes and volume groups to use, you can create them directly by following the Advanced configuration scenarios below.

      3. Advanced configuration

         First Scenario

         devices:
           - /dev/sda
           - /dev/sdb
         dedicated_devices:
           - /dev/sdx
           - /dev/sdy

         or

         Second Scenario

         devices:
           - /dev/sda
           - /dev/sdb
         dedicated_devices:
           - /dev/sdx
           - /dev/sdy
         bluestore_wal_devices:
           - /dev/nvme0n1
           - /dev/nvme0n2

         In the first scenario, there are two OSDs. The sda and sdb devices each have their own data segments and write-ahead logs. The additional dictionary dedicated_devices is used to isolate their databases, also known as block.db, on sdx and sdy, respectively.

         In the second scenario, another additional dictionary, bluestore_wal_devices, is used to isolate the write-ahead log on NVMe devices nvme0n1 and nvme0n2. Using the devices, dedicated_devices, and bluestore_wal_devices, options together, this allows you to isolate all components of an OSD onto separate devices. Laying out the OSDs like this can increase overall performance.

      4. Pre-created logical volumes

         First Scenario

         lvm_volumes:
           - data: data-lv1
             data_vg: data-vg1
             db: db-lv1
             db_vg: db-vg1
             wal: wal-lv1
             wal_vg: wal-vg1
           - data: data-lv2
             data_vg: data-vg2
             db: db-lv2
             db_vg: db-vg2
             wal: wal-lv2
             wal_vg: wal-vg2

         or

         Second Scenario

         lvm_volumes:
           - data: /dev/sdb
             db:    db-lv1
             db_vg: db-vg1
             wal: wal-lv1
             wal_vg: wal-vg1

         By default, Ceph uses Logical Volume Manager to create logical volumes on the OSD devices. In the Simple configuration and Advanced configuration examples above, Ceph creates logical volumes on the devices automatically. You can use previously created logical volumes with Ceph by specifying the lvm_volumes dictionary.

         In the first scenario, the data is placed on dedicated logical volumes, database, and WAL. You can also specify just data, data and WAL, or data and database. The data: line must specify the logical volume name where data is to be stored, and data_vg: must specify the name of the volume group the data logical volume is contained in. Similarly, db: is used to specify the logical volume the database is stored on and db_vg: is used to specify the volume group its logical volume is in. The wal: line specifies the logical volume the WAL is stored on and the wal_vg: line specifies the volume group that contains it.

         In the second scenario, the actual device name is set for the data: option, and doing so, does not require specifying the data_vg: option. You must specify the logical volume name and the volume group details for the BlueStore database and WAL devices.

         Important

         With lvm_volumes:, the volume groups and logical volumes must be created beforehand. The volume groups and logical volumes will not be created by ceph-ansible.

         Note

         If using all NVMe SSDs, then set osds_per_device: 2. For more information, see Configuring OSD Ansible settings for all NVMe Storage in the Red Hat Ceph Storage Installation Guide.

         Note

         After rebooting a Ceph OSD node, there is a possibility that the block device assignments will change. For example, sdc might become sdd. You can use persistent naming devices, such as the /dev/disk/by-path/ device path, instead of the traditional block device name.

6. For all deployments, bare-metal or in containers, create the Ansible inventory file and then open it for editing:

   [root@admin ~]# cd /usr/share/ceph-ansible/
   [root@admin ceph-ansible]# touch hosts

   Edit the hosts file accordingly.

   Note

   For information about editing the Ansible inventory location, see Configuring Ansible inventory location.

   1. Add a node under [grafana-server]. This role installs Grafana and Prometheus to provide real-time insights into the performance of the Ceph cluster. These metrics are presented in the Ceph Dashboard, which allows you to monitor and manage the cluster. The installation of the dashboard, Grafana, and Prometheus are required. You can colocate the metrics functions on the Ansible Administration node. If you do, ensure the system resources of the node are greater than than what is required for a stand alone metrics node.

      [grafana-server]
      GRAFANA-SERVER_NODE_NAME

   2. Add the monitor nodes under the [mons] section:

      [mons]
      MONITOR_NODE_NAME_1
      MONITOR_NODE_NAME_2
      MONITOR_NODE_NAME_3

   3. Add OSD nodes under the [osds] section:

      [osds]
      OSD_NODE_NAME_1
      OSD_NODE_NAME_2
      OSD_NODE_NAME_3

      Note

      You can add a range specifier ([1:10]) to the end of the node name, if the node names are numerically sequential. For example:

      [osds]
      example-node[1:10]

      Note

      For OSDs in a new installation, the default object store format is BlueStore.

   4. Optionally, in container deployments, colocate Ceph Monitor daemons with the Ceph OSD daemons on one node by adding the same node under the [mon] and [osd] sections. In the Additional Resources section below, see the link on colocating Ceph daemons for more information.

   5. Add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. This is colocating the Ceph Manager daemon with Ceph Monitor daemon.

      [mgrs]
      MONITOR_NODE_NAME_1
      MONITOR_NODE_NAME_2
      MONITOR_NODE_NAME_3

7. Optionally, if you want to use host specific parameters, for all deployments, bare-metal or in containers, create the host_vars directory with host files to include any parameters specific to hosts.

   1. Create the host_vars directory:

      [ansible@admin ~]$ mkdir /usr/share/ceph-ansible/host_vars

   2. Change to the host_vars directory:

      [ansible@admin ~]$ cd /usr/share/ceph-ansible/host_vars

   3. Create the host files. Use the host-name-short-name format for the name of the files, for example:

      [ansible@admin host_vars]$ touch tower-osd6

   4. Update the file with any host specific parameters, for example:

      1. In bare-metal deployments use the devices parameter to specify devices that the OSD nodes will use. Using devices is useful when OSDs use devices with different names or when one of the devices failed on one of the OSDs.

         devices:
             DEVICE_1
             DEVICE_2

         Example

         devices:
             /dev/sdb
             /dev/sdc

         Note

         When specifying no devices, set the osd_auto_discovery parameter to true in the group_vars/osds.yml file.

8. Optionally, for all deployments, bare-metal or in containers, you can create a custom CRUSH hierarchy using Ceph Ansible:

   1. Setup your Ansible inventory file. Specify where you want the OSD hosts to be in the CRUSH map’s hierarchy by using the osd_crush_location parameter. You must specify at least two CRUSH bucket types to specify the location of the OSD, and one bucket type must be host. By default, these include root, datacenter, room, row, pod, pdu, rack, chassis and host.

      Syntax

      [osds]
      CEPH_OSD_NAME osd_crush_location="{ 'root': ROOT_BUCKET_', 'rack': 'RACK_BUCKET', 'pod': 'POD_BUCKET', 'host': 'CEPH_HOST_NAME' }"

      Example

      [osds]
      ceph-osd-01 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'pod': 'monpod', 'host': 'ceph-osd-01' }"

   2. Edit the group_vars/osds.yml file, and set the crush_rule_config and create_crush_tree parameters to True. Create at least one CRUSH rule if you do not want to use the default CRUSH rules, for example:

      crush_rule_config: True
      crush_rule_hdd:
          name: replicated_hdd_rule
          root: root-hdd
          type: host
          class: hdd
          default: True
      crush_rules:
        - "{{ crush_rule_hdd }}"
      create_crush_tree: True

      If you are using faster SSD devices, then edit the parameters as follows:

      crush_rule_config: True
      crush_rule_ssd:
          name: replicated_ssd_rule
          root: root-ssd
          type: host
          class: ssd
          default: True
      crush_rules:
        - "{{ crush_rule_ssd }}"
      create_crush_tree: True

      Note

      The default CRUSH rules fail if both ssd and hdd OSDs are not deployed because the default rules now include the class parameter, which must be defined.

   3. Create pools, with created crush_rules in group_vars/clients.yml file:

      Example

      copy_admin_key: True
      user_config: True
      pool1:
        name: "pool1"
        pg_num: 128
        pgp_num: 128
        rule_name: "HDD"
        type: "replicated"
        device_class: "hdd"
      pools:
        - "{{ pool1 }}"

   4. View the tree:

      [root@mon ~]# ceph osd tree

   5. Validate the pools:

      [root@mon ~]# for i in $(rados lspools); do echo "pool: $i"; ceph osd pool get $i crush_rule; done
      
      pool: pool1
      crush_rule: HDD

9. For all deployments, bare-metal or in containers, log in with or switch to the ansible user.

   1. Create the ceph-ansible-keys directory where Ansible stores temporary values generated by the ceph-ansible playbook:

      [ansible@admin ~]$ mkdir ~/ceph-ansible-keys

   2. Change to the /usr/share/ceph-ansible/ directory:

      [ansible@admin ~]$ cd /usr/share/ceph-ansible/

   3. Verify that Ansible can reach the Ceph nodes:

      [ansible@admin ceph-ansible]$ ansible all -m ping -i hosts

10. Run the ceph-ansible playbook.

    1. Bare-metal deployments:

       [ansible@admin ceph-ansible]$ ansible-playbook site.yml -i hosts

    2. Container deployments:

       [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml -i hosts

       Note

       If you deploy Red Hat Ceph Storage to Red Hat Enterprise Linux Atomic Host hosts, use the --skip-tags=with_pkg option:

       [user@admin ceph-ansible]$ ansible-playbook site-container.yml --skip-tags=with_pkg -i hosts

       Note

       To increase the deployment speed, use the --forks option to ansible-playbook. By default, ceph-ansible sets forks to 20. With this setting, up to twenty nodes will be installed at the same time. To install up to thirty nodes at a time, run ansible-playbook --forks 30 PLAYBOOK FILE -i hosts. The resources on the admin node must be monitored to ensure they are not overused. If they are, lower the number passed to --forks.

11. Wait for the Ceph deployment to finish.

    Example output

    INSTALLER STATUS *******************************
    Install Ceph Monitor           : Complete (0:00:30)
    Install Ceph Manager           : Complete (0:00:47)
    Install Ceph OSD               : Complete (0:00:58)
    Install Ceph RGW               : Complete (0:00:34)
    Install Ceph Dashboard         : Complete (0:00:58)
    Install Ceph Grafana           : Complete (0:00:50)
    Install Ceph Node Exporter     : Complete (0:01:14)

12. Verify the status of the Ceph storage cluster.

    1. Bare-metal deployments:

       [root@mon ~]# ceph health
       HEALTH_OK

    2. Container deployments:

       Red Hat Enterprise Linux 7

       [root@mon ~]# docker exec ceph-mon-ID ceph health

       Red Hat Enterprise Linux 8

       [root@mon ~]# podman exec ceph-mon-ID ceph health

       Replace

       • ID with the host name of the Ceph Monitor node:

         Example

         [root@mon ~]# podman exec ceph-mon-mon0 ceph health
         HEALTH_OK

13. For all deployments, bare-metal or in containers, verify the storage cluster is functioning using rados.

    1. From a Ceph Monitor node, create a test pool with eight placement groups (PG):

       Syntax

       [root@mon ~]# ceph osd pool create POOL_NAME PG_NUMBER

       Example

       [root@mon ~]# ceph osd pool create test 8

    2. Create a file called hello-world.txt:

       Syntax

       [root@mon ~]# vim FILE_NAME

       Example

       [root@mon ~]# vim hello-world.txt

    3. Upload hello-world.txt to the test pool using the object name hello-world:

       Syntax

       [root@mon ~]# rados --pool POOL_NAME put OBJECT_NAME OBJECT_FILE_NAME

       Example

       [root@mon ~]# rados --pool test put hello-world hello-world.txt

    4. Download hello-world from the test pool as file name fetch.txt:

       Syntax

       [root@mon ~]# rados --pool POOL_NAME get OBJECT_NAME OBJECT_FILE_NAME

       Example

       [root@mon ~]# rados --pool test get hello-world fetch.txt

    5. Check the contents of fetch.txt:

       [root@mon ~]# cat fetch.txt
       "Hello World!"

       Note

       In addition to verifying the storage cluster status, you can use the ceph-medic utility to overall diagnose the Ceph Storage cluster. See the Installing and Using ceph-medic to Diagnose a Ceph Storage Cluster chapter in the Red Hat Ceph Storage 4 Troubleshooting Guide.

Procedure

1. Set osds_per_device: 2 in group_vars/osds.yml:

   osds_per_device: 2

2. List the NVMe devices under devices:

   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

3. The settings in group_vars/osds.yml will look similar to this example:

   osds_per_device: 2
   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

1. Add a new section [mdss] to the /etc/ansible/hosts file:

   [mdss]
   MDS_NODE_NAME1
   MDS_NODE_NAME2
   MDS_NODE_NAME3

   Replace MDS_NODE_NAME with the host names of the nodes where you want to install the Ceph Metadata servers.

   Alternatively, you can colocate the Metadata server with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Optionally, you can change the default variables.

   1. Create a copy of the group_vars/mdss.yml.sample file named mdss.yml:

      [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml

   2. Optionally, edit the parameters in mdss.yml. See mdss.yml for details.

4. As the ansible user, run the Ansible playbook:

   • Bare-metal deployments:

     [user@admin ceph-ansible]$ ansible-playbook site.yml --limit mdss -i hosts

   • Container deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit mdss -i hosts

5. After installing the Metadata servers, you can now configure them. For details, see the The Ceph File System Metadata Server chapter in the Red Hat Ceph Storage File System Guide.

1. Add a new section [clients] to the /etc/ansible/hosts file:

   [clients]
   CLIENT_NODE_NAME

   Replace CLIENT_NODE_NAME with the host name of the node where you want to install the ceph-client role.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Create a new copy of the clients.yml.sample file named clients.yml:

   [root@admin ceph-ansible ~]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Open the group_vars/clients.yml file, and uncomment the following lines:

   keys:
     - { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

   1. Replace client.test with the real client name, and add the client key to the client definition line, for example:

      key: "ADD-KEYRING-HERE=="

      Now the whole line example would look similar to this:

      - { name: client.test, key: "AQAin8tUMICVFBAALRHNrV0Z4MXupRw4v9JQ6Q==", caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

      Note

      The ceph-authtool --gen-print-key command can generate a new client key.

5. Optionally, instruct ceph-client to create pools and clients.

   1. Update clients.yml.

      • Uncomment the user_config setting and set it to true.
      • Uncomment the pools and keys sections and update them as required. You can define custom pools and client names altogether with the cephx capabilities.

   2. Add the osd_pool_default_pg_num setting to the ceph_conf_overrides section in the all.yml file:

      ceph_conf_overrides:
         global:
            osd_pool_default_pg_num: NUMBER

      Replace NUMBER with the default number of placement groups.

6. As the ansible user, run the Ansible playbook:

   1. Bare-metal deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit clients -i hosts

   2. Container deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit clients -i hosts

1. Add gateway hosts to the /etc/ansible/hosts file under the [rgws] section to identify their roles to Ansible. If the hosts have sequential naming, use a range, for example:

   [rgws]
   <rgw_host_name_1>
   <rgw_host_name_2>
   <rgw_host_name[3..10]>

2. Navigate to the Ansible configuration directory:

   [root@ansible ~]# cd /usr/share/ceph-ansible

3. Create the rgws.yml file from the sample file:

   [root@ansible ~]# cp group_vars/rgws.yml.sample group_vars/rgws.yml

4. Open and edit the group_vars/rgws.yml file. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key option:

   copy_admin_key: true

5. In the all.yml file, you MUST specify a radosgw_interface.

   radosgw_interface: <interface>

   Replace:

   • <interface> with the interface that the Ceph Object Gateway nodes listen to

   For example:

   radosgw_interface: eth0

   Specifying the interface prevents Civetweb from binding to the same IP address as another Civetweb instance when running multiple instances on the same host.

   For additional details, see the all.yml file.

6. Generally, to change default settings, uncomment the settings in the rgws.yml file, and make changes accordingly. To make additional changes to settings that are not in the rgws.yml file, use ceph_conf_overrides: in the all.yml file.

   ceph_conf_overrides:
       client.rgw.rgw1:
         rgw_override_bucket_index_max_shards: 16
         rgw_bucket_default_quota_max_objects: 1638400

   For advanced configuration details, see the Red Hat Ceph Storage 4 Ceph Object Gateway for Production guide. Advanced topics include:

   • Configuring Ansible Groups

   • Developing Storage Strategies. See the Creating the Root Pool, Creating System Pools, and Creating Data Placement Strategies sections for additional details on how create and configure the pools.

     See Bucket Sharding for configuration details on bucket sharding.

7. Run the Ansible playbook:

   Warning

   Do not run the Ansible playbook if you intend to set up multisite. Proceed to the Configuring multisite Ceph Object Gateways section to set up multisite.

   1. Bare-metal deployments:

      [user@admin ceph-ansible]$ ansible-playbook site.yml --limit rgws -i hosts

   2. Container deployments:

      [user@admin ceph-ansible]$ ansible-playbook site-container.yml --limit rgws -i hosts

Procedure

1. Create crush_rules in the group_vars/mons.yml file:

   Example

   crush_rule_config: true
   crush_rule_hdd:
       name: HDD
       root: default
       type: host
       class: hdd
       default: true
   crush_rule_ssd:
       name: SSD
       root: default
       type: host
       class: ssd
       default: true
   crush_rules:
         - "{{ crush_rule_hdd }}"
         - "{{ crush_rule_ssd }}"
   create_crush_tree: true

   Note

   If you are not using SSD or HDD devices in the cluster, do not define the crush_rules for that device.

2. Create pools, with created crush_rules in group_vars/clients.yml file.

   Example

   copy_admin_key: True
   user_config: True
   pool1:
     name: "pool1"
     pg_num: 128
     pgp_num: 128
     rule_name: "HDD"
     type: "replicated"
     device_class: "hdd"
   pools:
     - "{{ pool1 }}"

3. Sample the inventory file to assign roots to OSDs:

   Example

   [mons]
   mon1
   
   [osds]
   osd1 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'host': 'osd1' }"
   osd2 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'host': 'osd2' }"
   osd3 osd_crush_location="{ 'root': 'default', 'rack': 'rack2', 'host': 'osd3' }"
   osd4 osd_crush_location="{ 'root': 'default', 'rack': 'rack2', 'host': 'osd4' }"
   osd5 devices="['/dev/sda', '/dev/sdb']" osd_crush_location="{ 'root': 'default', 'rack': 'rack3', 'host': 'osd5' }"
   osd6 devices="['/dev/sda', '/dev/sdb']" osd_crush_location="{ 'root': 'default', 'rack': 'rack3', 'host': 'osd6' }"
   
   [mgrs]
   mgr1
   
   [clients]
   client1

4. View the tree.

   Syntax

   [root@mon ~]# ceph osd tree

   Example

   TYPE NAME
   
   root default
        rack rack1
           host osd1
                osd.0
                osd.10
           host osd2
                osd.3
                osd.7
                osd.12
        rack rack2
           host osd3
                osd.1
                osd.6
                osd.11
           host osd4
                osd.4
                osd.9
                osd.13
        rack rack3
            host osd5
                osd.2
                osd.8
            host osd6
                osd.14
                osd.15

5. Validate the pools.

   Example

   # for i in $(rados lspools);do echo "pool: $i"; ceph osd pool get $i crush_rule;done
   
   pool: pool1
   crush_rule: HDD

1. Create the nfss.yml file from the sample file:

   [root@ansible ~]# cd /usr/share/ceph-ansible/group_vars
   [root@ansible ~]# cp nfss.yml.sample nfss.yml

2. Add gateway hosts to the /etc/ansible/hosts file under an [nfss] group to identify their group membership to Ansible.

   [nfss]
   NFS_HOST_NAME_1
   NFS_HOST_NAME_2
   NFS_HOST_NAME[3..10]

   If the hosts have sequential naming, then you can use a range specifier, for example: [3..10].

3. Navigate to the Ansible configuration directory:

   [root@ansible ~]# cd /usr/share/ceph-ansible

4. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key setting in the /usr/share/ceph-ansible/group_vars/nfss.yml file:

   copy_admin_key: true

5. Configure the FSAL (File System Abstraction Layer) sections of the /usr/share/ceph-ansible/group_vars/nfss.yml file. Provide an export ID (NUMERIC_EXPORT_ID), S3 user ID (S3_USER), S3 access key (ACCESS_KEY) and secret key (SECRET_KEY):

   # FSAL RGW Config #
   
   ceph_nfs_rgw_export_id: NUMERIC_EXPORT_ID
   #ceph_nfs_rgw_pseudo_path: "/"
   #ceph_nfs_rgw_protocols: "3,4"
   #ceph_nfs_rgw_access_type: "RW"
   ceph_nfs_rgw_user: "S3_USER"
   ceph_nfs_rgw_access_key: "ACCESS_KEY"
   ceph_nfs_rgw_secret_key: "SECRET_KEY"

   Warning

   Access and secret keys are optional, and can be generated.

6. Run the Ansible playbook:

   1. Bare-metal deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit nfss -i hosts

   2. Container deployments:

      [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit nfss -i hosts