Chapter 4. Deploying Other Ceph Services on Dedicated Nodes

By default, the director deploys the Ceph MON and Ceph MDS services on the Controller nodes. This is suitable for small deployments. However, with larger deployments we advise that you deploy the Ceph MON and Ceph MDS services on dedicated nodes to improve the performance of your Ceph cluster. You can do this by creating a custom role for either one.

Note

For detailed information about custom roles, see Creating a New Role from the Advanced Overcloud Customization guide.

The director uses the following file as a default reference for all overcloud roles:

  • /usr/share/openstack-tripleo-heat-templates/roles_data.yaml

Copy this file to /home/stack/templates/ so you can add custom roles to it:

$ cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml /home/stack/templates/roles_data_custom.yaml

You invoke the /home/stack/templates/roles_data_custom.yaml file later during overcloud creation (Section 8.2, “Initiating Overcloud Deployment”). The following sub-sections describe how to configure custom roles for either Ceph MON and Ceph MDS services.

4.1. Creating a Custom Role and Flavor for the Ceph MON Service

This section describes how to create a custom role (named CephMon) and flavor (named ceph-mon) for the Ceph MON role. You should already have a copy of the default roles data file as described in Chapter 4, Deploying Other Ceph Services on Dedicated Nodes.

  1. Open the /home/stack/templates/roles_data_custom.yaml file.
  2. Remove the service entry for the Ceph MON service (namely, OS::TripleO::Services::CephMon) from under the Controller role.
  3. Add the OS::TripleO::Services::CephClient service to the Controller role:

    [...]
    - name: Controller # the 'primary' role goes first
      CountDefault: 1
      ServicesDefault:
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CephMds
        - OS::TripleO::Services::CephClient
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CephRbdMirror
        - OS::TripleO::Services::CephRgw
        - OS::TripleO::Services::CinderApi
    [...]
  4. At the end of roles_data_custom.yaml, add a custom CephMon role containing the Ceph MON service and all the other required node services. For example:

    - name: CephMon
      ServicesDefault:
        # Common Services
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::FluentdClient
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::Ntp
        - OS::TripleO::Services::ContainersLogrotateCrond
        - OS::TripleO::Services::SensuClient
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
        - OS::TripleO::Services::Tuned
        # Role-Specific Services
        - OS::TripleO::Services::CephMon
  5. Using the openstack flavor create command, define a new flavor named ceph-mon for this role:

    $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mon
    Note

    For more details about this command, run openstack flavor create --help.

  6. Map this flavor to a new profile, also named ceph-mon:

    $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mon" ceph-mon
    Note

    For more details about this command, run openstack flavor set --help.

  7. Tag nodes into the new ceph-mon profile:
$ ironic node-update UUID add properties/capabilities='profile:ceph-mon,boot_option:local'

See Section 3.3, “Manually Tagging the Nodes” for more details about tagging nodes. See also Tagging Nodes Into Profiles for related information on custom role profiles.

4.2. Creating a Custom Role and Flavor for the Ceph MDS Service

This section describes how to create a custom role (named CephMDS) and flavor (named ceph-mds) for the Ceph MDS role. You should already have a copy of the default roles data file as described in Chapter 4, Deploying Other Ceph Services on Dedicated Nodes.

  1. Open the /home/stack/templates/roles_data_custom.yaml file.
  2. Remove the service entry for the Ceph MDS service (namely, OS::TripleO::Services::CephMds) from under the Controller role:

    [...]
    - name: Controller # the 'primary' role goes first
      CountDefault: 1
      ServicesDefault:
        - OS::TripleO::Services::CACerts
        # - OS::TripleO::Services::CephMds 1
        - OS::TripleO::Services::CephMon
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CephRbdMirror
        - OS::TripleO::Services::CephRgw
        - OS::TripleO::Services::CinderApi
    [...]
    1
    Comment out this line. This same service will be added to a custom role in the next step.
  3. At the end of roles_data_custom.yaml, add a custom CephMDS role containing the Ceph MDS service and all the other required node services. For example:

    - name: CephMDS
      ServicesDefault:
        # Common Services
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::FluentdClient
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::Ntp
        - OS::TripleO::Services::ContainersLogrotateCrond
        - OS::TripleO::Services::SensuClient
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
        - OS::TripleO::Services::Tuned
        # Role-Specific Services
        - OS::TripleO::Services::CephMds
        - OS::TripleO::Services::CephClient 1
    1
    The Ceph MDS service requires the admin keyring, which can be set by either Ceph MON or Ceph Client service. As we are deploying Ceph MDS on a dedicated node (without the Ceph MON service), include the Ceph Client service on the role as well.
  4. Using the openstack flavor create command, define a new flavor named ceph-mds for this role:

    $ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 ceph-mds
    Note

    For more details about this command, run openstack flavor create --help.

  5. Map this flavor to a new profile, also named ceph-mds:

    $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="ceph-mds" ceph-mds
    Note

    For more details about this command, run openstack flavor set --help.

Tag nodes into the new ceph-mds profile:

$ ironic node-update UUID add properties/capabilities='profile:ceph-mds,boot_option:local'

See Section 3.3, “Manually Tagging the Nodes” for more details about tagging nodes. See also Tagging Nodes Into Profiles for related information on custom role profiles.