Chapter 3. Deploying Ceph services on dedicated nodes

By default, director deploys the Ceph MON and Ceph MDS services on the Controller nodes. This is suitable for small deployments. However, with larger deployments, Red Hat recommends that you deploy the Ceph MON and Ceph MDS services on dedicated nodes to improve the performance of your Ceph cluster. Create a custom role for services that you want to isolate on dedicated nodes.

Note

For more information about custom roles, see Creating a New Role in the Advanced Overcloud Customization guide.

Director uses the following file as a default reference for all overcloud roles:

  • /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

3.1. Creating a custom roles file

To create a custom role file, complete the following steps:

Procedure

  1. Make a copy of the roles_data.yaml file in /home/stack/templates/ so that you can add custom roles: 

    $ cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml /home/stack/templates/roles_data_custom.yaml
  2. Include the new custom role file in the openstack overcloud deploy command.

3.2. Creating a custom role and flavor for the Ceph MON service

Create a custom role CephMon and flavor ceph-mon for the Ceph MON role.

Prerequisites

Procedure

  1. Open the /home/stack/templates/roles_data_custom.yaml file.
  2. Remove the service entry for the Ceph MON service, OS::TripleO::Services::CephMon, from the Controller role.

  3. Add the OS::TripleO::Services::CephClient service to the Controller role: 

    [...]
- name: Controller # the 'primary' role goes first
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephMds
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephRbdMirror
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CinderApi
[...]

  4. At the end of the roles_data_custom.yaml file, add a custom CephMon role that contains the Ceph MON service and all the other required node services: 

    - name: CephMon
  ServicesDefault:
    # Common Services
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
    # Role-Specific Services
    - OS::TripleO::Services::CephMon

  5. Enter the openstack flavor create command to define a new flavor named ceph-mon for the CephMon role: 

    $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mon
    Note

    For more information about this command, enter: openstack flavor create --help.

  6. Map this flavor to a new profile, also named ceph-mon: 

    $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mon" ceph-mon
    Note

    For more information about this command, enter openstack flavor set --help.

  7. Tag nodes into the new ceph-mon profile: 

    $ openstack baremetal node set --property capabilities='profile:ceph-mon,boot_option:local' UUID

  8. Add the following configuration to the node-info.yaml file to associate the ceph-mon flavor with the CephMon role: 

    parameter_defaults:
  OvercloudCephMonFlavor: CephMon
  CephMonCount: 3

For more information about tagging nodes, see Section 2.4, “Manually tagging nodes into profiles”. For more information about custom role profiles, see Tagging Nodes Into Profiles.

3.3. Creating a custom role and flavor for the Ceph MDS service

Complete the following steps to create a custom role CephMDS and flavor ceph-mds for the Ceph MDS role. You must already have a copy of the default roles data file as described in Chapter 3, Deploying Ceph services on dedicated nodes.

Procedure

  1. Open the /home/stack/templates/roles_data_custom.yaml file.

  2. Remove the service entry for the Ceph MDS service, OS::TripleO::Services::CephMds, from the Controller role: 

    [...]
- name: Controller # the 'primary' role goes first
  CountDefault: 1
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    # - OS::TripleO::Services::CephMds 1
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephRbdMirror
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CinderApi
[...]
    1
    Comment out this line. In the next step, you add this service to the new custom role.

  3. At the end of the roles_data_custom.yaml file, add a custom CephMDS role that contains the Ceph MDS service and all the other required node services: 

    - name: CephMDS
  ServicesDefault:
    # Common Services
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
    # Role-Specific Services
    - OS::TripleO::Services::CephMds
    - OS::TripleO::Services::CephClient 1
    1
    The Ceph MDS service requires the admin keyring, which you can set with either the Ceph MON or Ceph Client service. If you deploy Ceph MDS on a dedicated node without the Ceph MON service, you must also include the Ceph Client service in the new CephMDS role.

  4. Enter the openstack flavor create command to define a new flavor named ceph-mds for this role: 

    $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mds
    Note

    For more information about this command, enter openstack flavor create --help.

  5. Map the new ceph-mds flavor to a new profile, also named ceph-mds: 

    $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mds" ceph-mds
    Note

    For more information about this command, enter openstack flavor set --help.

  6. Tag nodes into the new ceph-mds profile: 

    $ openstack baremetal node set --property capabilities='profile:ceph-mds,boot_option:local' UUID

For more information about tagging nodes, see Section 2.4, “Manually tagging nodes into profiles”. For more information about custom role profiles, see Tagging Nodes Into Profiles.