The Ceph OSDs use this network to balance data according to the replication policy. This private network only needs to be accessed by the OSDs. In a hyper-converged deployment, the compute role needs access to this network. Thus, Chapter 5, Define the Overcloud, describes how to modify nic-configs/compute-nics.yaml to ensure that compute nodes are deployed with a connection to this network.

The Heat Provider that Red Hat OpenStack Platform director uses to define this network can be referenced with the following:

 OS::TripleO::Network::StorageMgmt: /usr/share/openstack-tripleo-heat-templates/network/storage_mgmt.yaml

The Ceph monitor nodes are accessed via this network. The Heat Provider that Red Hat OpenStack Platform director uses to define this network can be referenced with the following:

 OS::TripleO::Network::Storage: /usr/share/openstack-tripleo-heat-templates/network/storage.yaml

Red Hat OpenStack Platform director uses the external network to download software updates for the overcloud, and the cloud operator uses this network to access director to manage the overcloud.

The Controllers use the external network to route traffic to the Internet for tenant services that are externally connected via reserved floating IPs. Overcloud users use the external network to access the overcloud.

The Compute nodes do not need to be directly connected to the external network, as their instances communicate via the Tenant network to the Controllers who then route external traffic on their behalf to the external network.

The Heat Provider that Red Hat OpenStack Platform director uses to define this network can be referenced with the following:

 OS::TripleO::Network::External: /usr/share/openstack-tripleo-heat-templates/network/external.yaml

OpenStack provides both public facing and private API endpoints. This is an isolated network for the private endpoints.

The Heat Provider that Red Hat OpenStack Platform director uses to define this network can be referenced with the following:

 OS::TripleO::Network::InternalApi: /usr/share/openstack-tripleo-heat-templates/network/internal_api.yaml

OpenStack tenants create private networks implemented by VLAN or VXLAN on this network.

The Heat Provider that Red Hat OpenStack Platform director uses to define this network can be referenced with the following:

 OS::TripleO::Network::Tenant: /usr/share/openstack-tripleo-heat-templates/network/tenant.yaml

Red Hat OpenStack Platform director serves DHCP and PXE services from this network to install the operating system and other software on the overcloud nodes from baremetal. Red Hat OpenStack Platform director uses this network to manage the overcloud nodes, and the cloud operator uses it to access the overcloud nodes directly by ssh if necessary. The overcloud nodes must be configured to PXE boot from this network provisioning.

Figure 3.1. Figure 1. Network Separation Diagram

By default, Ironic images the first block device, idenified a /dev/sda, with the operating system during deployment. This section covers how to change the block device to be imaged, known as the root device, by using Root Device Hints.

The Compute/OSD servers used for this reference architecture have the following hard disks with the following device file names as seen by the operating system:

The RAID1 pair hosts the OS, while the twelve larger drives are configured as OSDs that journal to the SSDs. Since /dev/sda should be used for an OSD, Ironic needs to store which root device it should use instead of the default.

After introspection, Ironic stores the WWN and size of each server’s block device. Since the RAID1 pair is both the smallest disk and the disk that should be used for the root device, the openstack baremetal configure boot command may be run a second time, after introspection, as below:

 openstack baremetal configure boot --root-device=smallest

The above makes Ironic find the WWN of the smallest disk and then store a directive in its database to use that WWN for the root device when the server is deployed. Ironic does this for every server in its database. To verify that the directive was set for any particular server, run a command like the following:

[stack@hci-director ~]$ openstack baremetal node show r730xd_u33 | grep wwn
| properties             | {u'cpu_arch': u'x86_64', u'root_device': {u'wwn': u'0x614187704e9c7700'}, u'cpus': u'56', u'capabilities': u'node:osd-compute-2,cpu_hugepages:true,cpu_txt:true,boot_option:local,cpu_aes:true,cpu_vt:true,cpu_hugepages_1g:true', u'memory_mb': u'262144', u'local_gb': 277}                          |
[stack@hci-director ~]$

In the above example u’root_device': {u’wwn': u'0x614187704e9c7700'} indicates that the root device is set to a specific WWN. The same command produces a similar result for each server. The server may be referred to by its name, as in the above example, but if the server does not have a name, then the UUID is used.

For the hardware used in this reference architecture, the size was a simple way to tell Ironic how to set the root device. For other hardware, other root device hints may be set using the vendor or model. If necessary, these values, in addition to the WWN and serial number, may be downloaded directly from Ironic’s Swift container and be explicitly to set the root device for each node. An example of how to do this may be found in section 5.4. Defining the Root Disk for Nodes of the Red Hat document Director Installation and Usage. If the root device of each node needs to be set explicitly, then a script may be written to automate setting this value for a large deployment. Though in the example above, a simple root device hint abstracts this automation so that Ironic may handle it, even for a large number of nodes.

Create a new file in ~/custom-templates called network.yaml and add content to this file.

The resource_registry section contains references network configuration for the controller/monitor and compute/OSD nodes. The first three lines of network.yaml contains the following:

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/custom-templates/nic-configs/controller-nics.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/custom-templates/nic-configs/compute-nics.yaml

The controller-nics.yaml and compute-nics.yaml files in the nic-configs directory above are created in the next section. Add the above to reference the empty files in the meantime.

Within the network.yaml, add the following parameters:

parameter_defaults:
  NeutronBridgeMappings: 'datacentre:br-ex,tenant:br-tenant'
  NeutronNetworkType: 'vxlan'
  NeutronTunnelType: 'vxlan'
  NeutronExternalNetworkBridge: "''"

The above defines the bridge mappings associated to the logical networks and enables tenants to use VXLAN.

The Heat templates referenced in the resource_registry require parameters to define the specifics of each network. For example, the IP address range and VLAN ID for the storage network to be created. Under the parameters_defaults section to network.yaml, define the parameters for each network. The value of the parameters should be based on the networks that OpenStack deploys. For this reference architecture, the parameter values may be found in the Network section of Appendix: Environment Details. These values are then supplied to the parameters_defaults section as follows:

  # Internal API used for private OpenStack Traffic
  InternalApiNetCidr: 192.168.2.0/24
  InternalApiAllocationPools: [{'start': '192.168.2.10', 'end': '192.168.2.200'}]
  InternalApiNetworkVlanID: 4049

  # Tenant Network Traffic - will be used for VXLAN over VLAN
  TenantNetCidr: 192.168.3.0/24
  TenantAllocationPools: [{'start': '192.168.3.10', 'end': '192.168.3.200'}]
  TenantNetworkVlanID: 4050

  # Public Storage Access - e.g. Nova/Glance <--> Ceph
  StorageNetCidr: 172.16.1.0/24
  StorageAllocationPools: [{'start': '172.16.1.10', 'end': '172.16.1.200'}]
  StorageNetworkVlanID: 4046

  # Private Storage Access - i.e. Ceph background cluster/replication
  StorageMgmtNetCidr: 172.16.2.0/24
  StorageMgmtAllocationPools: [{'start': '172.16.2.10', 'end': '172.16.2.200'}]
  StorageMgmtNetworkVlanID: 4047

  # External Networking Access - Public API Access
  ExternalNetCidr: 10.19.137.0/21
  # Leave room for floating IPs in the External allocation pool (if required)
  ExternalAllocationPools: [{'start': '10.19.139.37', 'end': '10.19.139.48'}]
  # Set to the router gateway on the external network
  ExternalInterfaceDefaultRoute: 10.19.143.254

  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.1.1
  # The IP address of the EC2 metadata server. Generally the IP of the Undercloud
  EC2MetadataIp: 192.168.1.1
  # Define the DNS servers (maximum 2) for the Overcloud nodes
  DnsServers: ["10.19.143.247","10.19.143.248"]

For more information on the above directives see section 6.2, Isolating Networks, of the Red Hat document Advanced Overcloud Customization.

Within the network.yaml file, references were made to controller and compute Heat templates which need to be created in ~/custom-templates/nic-configs/. Complete the following steps to create these files:

 mkdir ~/custom-templates/nic-configs

Red Hat OpenStack Platform director contains a directory of network interface configuration templates for the following four scenarios:

$ ls /usr/share/openstack-tripleo-heat-templates/network/config/
bond-with-vlans  multiple-nics  single-nic-linux-bridge-vlans  single-nic-vlans
$

In this reference architecture, VLANs are trunked onto a single NIC. The following is creates the compute.yaml and controller.yaml files:

 cp /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-vlans/compute.yaml ~/custom-templates/nic-configs/compute-nics.yaml

 cp /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-vlans/controller.yaml ~/custom-templates/nic-configs/controller-nics.yaml

Modify ~/custom-templates/nic-configs/controller-nics.yaml based on the hardware that hosts the controller as described in 6.2. Creating a Network Environment File of the Red Hat document Advanced Overcloud Customization.

Modify ~/custom-templates/nic-configs/compute-nics.yaml as described in the previous step. Extend the provided template to include the StorageMgmtIpSubnet and StorageMgmtNetworkVlanID attributes of the storage management network. When defining the interface entries of the storage network, consider setting the MTU to 9000 (jumbo frames) for improved storage performance. An example of these additions to compute-nics.yaml includes the following:

             -
               type: interface
               name: em2
               use_dhcp: false
               mtu: 9000
             -
               type: vlan
               device: em2
               mtu: 9000
               use_dhcp: false
               vlan_id: {get_param: StorageMgmtNetworkVlanID}
               addresses:
                 -
                   ip_netmask: {get_param: StorageMgmtIpSubnet}
             -
               type: vlan
               device: em2
               mtu: 9000
               use_dhcp: false
               vlan_id: {get_param: StorageNetworkVlanID}
               addresses:
                 -
                   ip_netmask: {get_param: StorageIpSubnet}

Complete versions of compute-nics.yaml and controller-nics.yaml, as used in this reference architecture, may be found in Appendix: Custom Resources and Parameters. They may also be found online. See the Appendix G, GitHub Repository of Example Files Appendix for more details.

By default the overcloud consists of five roles: Controller, Compute, BlockStorage, ObjectStorage, and CephStorage. Each role consists of a list of services. As of Red Hat OpenStack Platform 10, the services that are deployed per role may be seen in /usr/share/openstack-tripleo-heat-templates/roles_data.yaml.

It is possible to make a copy of roles_data.yaml and then define a new role within it consisting of any mix of available services found under other roles. This reference architecture follows this procedure to create a new role called OsdCompute. For more information about Composable Roles themselves, see the Composable Roles section of the Red Hat document Advanced Overcloud Customization.

Copy the roles data file to the custom templates directory for modification.

cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml ~/custom-templates/custom-roles.yaml

Edit ~/custom-templates/custom-roles.yaml to add the following to the bottom of the file which defines a new role called OsdCompute.

- name: OsdCompute
  CountDefault: 0
  HostnameFormatDefault: '%stackname%-osd-compute-%index%'
  ServicesDefault:
    - OS::TripleO::Services::CephOSD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::NovaCompute
    - OS::TripleO::Services::NovaLibvirt
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::ComputeNeutronCorePlugin
    - OS::TripleO::Services::ComputeNeutronOvsAgent
    - OS::TripleO::Services::ComputeCeilometerAgent
    - OS::TripleO::Services::ComputeNeutronL3Agent
    - OS::TripleO::Services::ComputeNeutronMetadataAgent
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::NeutronSriovAgent
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

The list of services under ServicesDefault is identical to the list of services under the Compute role, except the CephOSD service has been added to the list. The CountDefault of 0 ensures that no nodes from this new role are deployed unless explicitly requested and the HostnameFormatDefault defines what each node should be called when deployed, e.g. overcloud-osd-compute-0, overcloud-osd-compute-1, etc.

In the Chapter 7, Deployment section, the new ~/custom-templates/custom-roles.yaml, which contains the above, is passed to the openstack overcloud deploy command.

To set the resource registry for Ceph, add a resource_registry section to which includes a first-boot template and a post-deploy template to ceph.yaml.

resource_registry:
  OS::TripleO::NodeUserData: /home/stack/custom-templates/first-boot-template.yaml
  OS::TripleO::NodeExtraConfigPost: /home/stack/custom-templates/post-deploy-template.yaml

The first-boot-template.yaml and post-deploy-template.yaml files above are used to configure Ceph during the deployment and are created in the next subsection.

heat_template_version: 2014-10-16

description: >
  Wipe and convert all disks to GPT (except the disk containing the root file system)

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: wipe_disk}

  wipe_disk:
    type: OS::Heat::SoftwareConfig
    properties:
      config: {get_file: wipe-disk.sh}

outputs:
  OS::stack_id:
    value: {get_resource: userdata}

#!/usr/bin/env bash
if [[ `hostname` = *"ceph"* ]] || [[ `hostname` = *"osd-compute"* ]]
then
  echo "Number of disks detected: $(lsblk -no NAME,TYPE,MOUNTPOINT | grep "disk" | awk '{print $1}' | wc -l)"
  for DEVICE in `lsblk -no NAME,TYPE,MOUNTPOINT | grep "disk" | awk '{print $1}'`
  do
    ROOTFOUND=0
    echo "Checking /dev/$DEVICE..."
    echo "Number of partitions on /dev/$DEVICE: $(expr $(lsblk -n /dev/$DEVICE | awk '{print $7}' | wc -l) - 1)"
    for MOUNTS in `lsblk -n /dev/$DEVICE | awk '{print $7}'`
    do
      if [ "$MOUNTS" = "/" ]
      then
        ROOTFOUND=1
      fi
    done
    if [ $ROOTFOUND = 0 ]
    then
      echo "Root not found in /dev/${DEVICE}"
      echo "Wiping disk /dev/${DEVICE}"
      sgdisk -Z /dev/${DEVICE}
      sgdisk -g /dev/${DEVICE}
    else
      echo "Root found in /dev/${DEVICE}"
    fi
  done
fi

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json

resources:

  ExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      inputs:
        - name: OSD_NUMA_INTERFACE
      config: |
        #!/usr/bin/env bash
        {
        echo "TODO: pin OSDs to the NUMA node of $OSD_NUMA_INTERFACE"
        } 2>&1 > /root/post_deploy_heat_output.txt

  ExtraDeployments:
    type: OS::Heat::SoftwareDeployments
    properties:
      servers: {get_param: servers}
      config: {get_resource: ExtraConfig}
      input_values:
        OSD_NUMA_INTERFACE: 'em2'
      actions: ['CREATE']

[root@overcloud-osd-compute-2 ~]# cat /root/post_deploy_heat_output.txt
TODO: pin OSDs to the NUMA node of em2
[root@overcloud-osd-compute-2 ~]#

Add a parameter defaults section in ~/custom-templates/ceph.yaml under the resource registry defined in the previous subsection.

parameter_defaults:
  ExtraConfig:
    ceph::profile::params::osd_pool_default_pg_num: 256
    ceph::profile::params::osd_pool_default_pgp_num: 256
    ceph::profile::params::osd_pool_default_size: 3
    ceph::profile::params::osd_pool_default_min_size: 2
    ceph::profile::params::osd_recovery_op_priority: 2
  OsdComputeExtraConfig:
    ceph::profile::params::osd_journal_size: 5120

    ceph::profile::params::osds:
      '/dev/sda':
        journal: '/dev/sdm'
      '/dev/sdb':
        journal: '/dev/sdm'
      '/dev/sdc':
        journal: '/dev/sdm'
      '/dev/sdd':
        journal: '/dev/sdm'
      '/dev/sde':
        journal: '/dev/sdn'
      '/dev/sdf':
        journal: '/dev/sdn'
      '/dev/sdg':
        journal: '/dev/sdn'
      '/dev/sdh':
        journal: '/dev/sdn'
      '/dev/sdi':
        journal: '/dev/sdo'
      '/dev/sdj':
        journal: '/dev/sdo'
      '/dev/sdk':
        journal: '/dev/sdo'
      '/dev/sdl':
        journal: '/dev/sdo'

Create the file ~/custom-templates/layout.yaml and add the following to it:

resource_registry:

  OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml
  OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml

  OS::TripleO::OsdCompute::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::OsdCompute::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml
  OS::TripleO::OsdCompute::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::OsdCompute::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml

The above defines the ports that are used by the Controller role and OsdCompute role. Each role has four lines of very similar template includes, but the Controller lines override what is already defined by the default for the Controller role, in order to get its IPs from a list defined later in this file. In contrast, the OsdCompute role has no default ports to override, as it is a new custom role. The ports that it uses are defined in this Heat template. In both cases the IPs that each port receives are defined in a list under ControllerIPs or OsdComputeIPs which will be added to the layout.yaml file in the next section.

This subsection defines for each node type, how many of which should be deployed. Under the resource_registry of ~/custom-templates/layout.yaml, add the following:

parameter_defaults:
  NtpServer: 10.5.26.10

  ControllerCount: 3
  ComputeCount: 0
  CephStorageCount: 0
  OsdComputeCount: 3

In prior versions of Red Hat OpenStack Platform, the above parameters would have been passed through the command line, e.g. --ntp-server can now be passed in a Heat template as NtpServer as in the example above.

The above indicates that three Controller nodes are deployed and that three OsdCompute nodes are deployed. In Section 8.2, “Adding Compute/Red Hat Ceph Storage Nodes” the value for OsdComputeCount is set to four to deploy an additional OsdCompute node. For this deployment, it is necessary to set the number of Compute nodes to zero because the CountDefault for a Compute node is 1, as can be verified by reading /usr/share/openstack-tripleo-heat-templates/roles_data.yaml. It is not necessary set the CephStorageCount to 0, because none are deployed by default. However, this parameter is included in this example to demonstrate how to add separate Ceph OSD servers that do not offer Nova compute services, and similarly the ComputeCount could be changed to offer Nova compute servers that do not offer Ceph OSD services. The process to mix hyper-converged Ceph/OSD nodes with non-hyper-converged Ceph or OSD nodes in a single deployment, is to increase these counts and define the properties of either standard role.

In the Section 4.2, “Register and Introspect Hardware” section, each node is assigned a capabilities profile of either controller-X or osd-compute-Y, where X or Y was the number of the physical node. These labels are used by the scheduler hints feature in Red Hat OpenStack Platform director to define specific node placement and ensure that a particular physical node is always be deployed to the overcloud with the same assignemnt; e.g. to ensure that the server in rack U33 is always osd-compute-2.

Append the following to layout.yaml under the parameter_defaults stanza to implement the predictable node palcement described above.

  ControllerSchedulerHints:
    'capabilities:node': 'controller-%index%'
  NovaComputeSchedulerHints:
    'capabilities:node': 'compute-%index%'
  CephStorageSchedulerHints:
    'capabilities:node': 'ceph-storage-%index%'
  OsdComputeSchedulerHints:
    'capabilities:node': 'osd-compute-%index%'

Add the following to layout.yaml under the parameter_defaults stanza above to ensure that each node gets a specific IP.

  ControllerIPs:
    internal_api:
      - 192.168.2.200
      - 192.168.2.201
      - 192.168.2.202
    tenant:
      - 192.168.3.200
      - 192.168.3.201
      - 192.168.3.202
    storage:
      - 172.16.1.200
      - 172.16.1.201
      - 172.16.1.202
    storage_mgmt:
      - 172.16.2.200
      - 172.16.2.201
      - 172.16.2.202

  OsdComputeIPs:
    internal_api:
      - 192.168.2.203
      - 192.168.2.204
      - 192.168.2.205
    tenant:
      - 192.168.3.203
      - 192.168.3.204
      - 192.168.3.205
    storage:
      - 172.16.1.203
      - 172.16.1.204
      - 172.16.1.205
    storage_mgmt:
      - 172.16.2.203
      - 172.16.2.204
      - 172.16.2.205

The above specifies that the following predictable IP assignment will happen for each deploy:

and so on for the controller nodes and:

and so on for the osd-compute nodes.

For more information on assigning node specific identification, see section 7.1. Assigning Specific Node IDs of the Red Hat document Advanced Overcloud Customization.

Nova’s reserved_host_memory_mb is the amount of memory in MB to reserve for the host. If a node is dedicated only to offering compute services, then this value should be set to maximize the number of running guests. However, on a system that must also support Ceph OSDs, this value needs to be increased so that Ceph has access to the memory that it needs.

To determine the reserved_host_memory_mb for a hyper-converged node, assume that each OSD consumes 3GB of RAM. Given a node with 256GB of RAM and 10 OSDs, 30GB of RAM is used for Ceph and 226GB of RAM is available for Nova. If the average guest uses the m1.small flavor, which uses 2GB of RAM per guest, then the overall system could host 113 such guests. However there is an additional overhead to account for per guest for the hypervisor. Assume this overhead is a half GB. With this overhead taken into account, the maximum number of 2GB guests that could be run would be 226GB divided by 2.5GB of RAM, which is approximately 90 virtual guests.

Given this number of guests and the number of OSDs, the amount of memory to reserve that Nova cannot use would be the amount of guests times their overhead plus the amount of OSDs times the amount of RAM that each OSD should have. In other words, (90*0.5) + (10*3), which is 75GB. Nova expects this value in MB and thus 75000 would be provided to the nova.conf.

These ideas may be expressed mathematically in the following Python code:

left_over_mem = mem - (GB_per_OSD * osds)
number_of_guests = int(left_over_mem /
                       (average_guest_size + GB_overhead_per_guest))
nova_reserved_mem_MB = MB_per_GB * (
                        (GB_per_OSD * osds) +
                        (number_of_guests * GB_overhead_per_guest))

The above is from the Nova Memory and CPU Calculator, which is covered in a future section of this paper.

Nova’s cpu_allocation_ratio is used by the Nova scheduler when choosing compute nodes to run guests. If the ratio has the default of 16:1 and the number of cores on a node, also known as vCPUs, is 56, then the Nova scheduler may schedule enough guests to consume 896 vCPUs before it considers the node unable to handle any more guests. Because the Nova scheduler does not take into account the CPU needs of Ceph OSD services running on the same node, the cpu_allocation_ratio should be modified so that Ceph has the CPU resources it needs to operate effectively without those CPU resources being given to Nova.

To determine the cpu_allocation_ratio for a hyper-converged node, assume that at least one core is used by each OSD (unless the workload is IO intensive). Given a node with 56 cores and 10 OSDs, that leaves 46 cores for Nova. If each guest uses 100% of the CPU that it is given, then the ratio should be the number of guest vCPUs divided by the number of cores; that is, 46 divided by 56, or 0.8. However, because guests don’t usually consume 100% of their CPUs, the ratio should be raised by taking the anticipated percentage into account when determining the number of required guest vCPUs. So, if only 10%, or 0.1, of a vCPU is used by a guest, then the number of vCPUs for guests is 46 divided by 0.1, or 460. When this value is divided by the number of cores, 56, the ratio increases to approximately 8.

These ideas may be expressed mathematically in the following Python code:

 cores_per_OSD = 1.0
 average_guest_util = 0.1 # 10%
 nonceph_cores = cores - (cores_per_OSD * osds)
 guest_vCPUs = nonceph_cores / average_guest_util
 cpu_allocation_ratio = guest_vCPUs / cores

The above is from the Nova Memory and CPU Calculator covered in the next section.

The formulas covered above are in a script called nova_mem_cpu_calc.py, which is available in Appendix: Nova Memory and CPU Calculator. It takes the following ordered parameters as input:

It prints as output a recommendation for how to set the nova.confreserved_host_memory_mb and cpu_allocation_ratio to favor stability of a hyper-converged deployment. When the numbers from the example discussed in the previous section are provided to the script, it returns the following results.

$ ./nova_mem_cpu_calc.py 256 56 10 2 1.0
Inputs:
- Total host RAM in GB: 256
- Total host cores: 56
- Ceph OSDs per host: 10
- Average guest memory size in GB: 2
- Average guest CPU utilization: 100%

Results:
- number of guests allowed based on memory = 90
- number of guest vCPUs allowed = 46
- nova.conf reserved_host_memory = 75000 MB
- nova.conf cpu_allocation_ratio = 0.821429

Compare "guest vCPUs allowed" to "guests allowed based on memory" for actual guest count
$

The amount of possible guests is bound by the limitations of either the CPU or the memory of the overcloud. In the example above, if each guest is using 100% of its CPU and there are only 46 vCPUs available, then it is not possible to launch 90 guests, even though there is enough memory to do so. If the anticipated guest CPU utilization decreases to only 10%, then the number of allowable vCPUs increases along with the cpu_allocation_ratio.

$ ./nova_mem_cpu_calc.py 256 56 10 2 0.1
Inputs:
- Total host RAM in GB: 256
- Total host cores: 56
- Ceph OSDs per host: 10
- Average guest memory size in GB: 2
- Average guest CPU utilization: 10%

Results:
- number of guests allowed based on memory = 90
- number of guest vCPUs allowed = 460
- nova.conf reserved_host_memory = 75000 MB
- nova.conf cpu_allocation_ratio = 8.214286

Compare "guest vCPUs allowed" to "guests allowed based on memory" for actual guest count
$

After determining the desired values of the reserved_host_memory_mb and cpu_allocation_ratio, proceed to the next section to apply the new settings.

Create the new file ~/custom-templates/compute.yaml containing the following:

parameter_defaults:
  ExtraConfig:
    nova::compute::reserved_host_memory: 75000
    nova::cpu_allocation_ratio: 8.2

In the above example ExtraConfig is used to change the amount of memory that the Nova compute service reserves in order to protect both the Ceph OSD service and the host itself. Also, in the above example, ExtraConfig is used to change the Nova CPU allocation ratio of the Nova scheduler service so that it does not allocate any of the CPUs that the Ceph OSD service uses.

To verify that the reserved host memory and CPU allocation ratio configuration changes were applied after Chapter 7, Deployment, ssh into any of the OsdCompute nodes and look for the configuration change in the nova.conf.

[root@overcloud-osd-compute-0 ~]# grep reserved_host_memory /etc/nova/nova.conf
reserved_host_memory_mb=75000
[root@overcloud-osd-compute-0 ~]#

[root@overcloud-osd-compute-0 ~]# grep cpu_allocation_ratio /etc/nova/nova.conf
cpu_allocation_ratio=8.2
[root@overcloud-osd-compute-0 ~]#

The Overcloud workload may vary over time so it is likely that the reserved_host_memory and cpu_allocation_ratio will need to be changed. To do so after Chapter 7, Deployment, simply update the the values in compute.yaml and re-run the deployment command covered in Chapter 7, Deployment. More details on overcloud updates are in the Section 8.1, “Configuration Updates” section.

In the Section 5.4, “Ceph Configuration” section, a post-deploy-template was added to the resource registry of ceph.yaml. That post deploy template originally contained only the following:

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json

resources:

  ExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      inputs:
        - name: OSD_NUMA_INTERFACE
      config: |
        #!/usr/bin/env bash
        {
        echo "TODO: pin OSDs to the NUMA node of $OSD_NUMA_INTERFACE"
        } 2>&1 > /root/post_deploy_heat_output.txt

  ExtraDeployments:
    type: OS::Heat::SoftwareDeployments
    properties:
      servers: {get_param: servers}
      config: {get_resource: ExtraConfig}
      input_values:
        OSD_NUMA_INTERFACE: 'em2'
      actions: ['CREATE']

The next two subsections will update the above file.

OSD_NUMA_INTERFACE: 'em2'

      config: |
        #!/usr/bin/env bash
        {
        echo "TODO: pin OSDs to the NUMA node of $OSD_NUMA_INTERFACE"
        } 2>&1 > /root/post_deploy_heat_output.txt

      config: {get_file: numa-systemd-osd.sh}

[root@overcloud-osd-compute-0 ~]# ./numa-systemd-osd.sh
changed: --set /usr/lib/systemd/system/ceph-osd@.service Service ExecStart '/usr/bin/numactl -N 0 --preferred=0 /usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph'

Status of OSD 1 before unit file update

* ceph-osd@1.service - Ceph object storage daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-12-16 02:50:02 UTC; 11min ago
 Main PID: 83488 (ceph-osd)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
           └─83488 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

Dec 16 02:50:01 overcloud-osd-compute-0.localdomain systemd[1]: Starting Ceph object storage daemon...
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain ceph-osd-prestart.sh[83437]: create-or-move update...
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain systemd[1]: Started Ceph object storage daemon.
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: starting osd.1 at :/0 osd_data /v...l
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: 2016-12-16 02:50:02.544592 7fecba...}
Dec 16 03:01:19 overcloud-osd-compute-0.localdomain systemd[1]: [/usr/lib/systemd/system/ceph-osd@.s...e'
Hint: Some lines were ellipsized, use -l to show in full.

Restarting OSD 1...

Status of OSD 1 after unit file update

* ceph-osd@1.service - Ceph object storage daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-12-16 03:01:21 UTC; 7ms ago
  Process: 89472 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
 Main PID: 89521 (numactl)
   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@1.service
           └─89521 /usr/bin/numactl -N 0 --preferred=0 /usr/bin/ceph-osd -f --cluster ceph --id 1 --se...

Dec 16 03:01:21 overcloud-osd-compute-0.localdomain systemd[1]: Starting Ceph object storage daemon...
Dec 16 03:01:21 overcloud-osd-compute-0.localdomain ceph-osd-prestart.sh[89472]: create-or-move update...
Dec 16 03:01:21 overcloud-osd-compute-0.localdomain systemd[1]: Started Ceph object storage daemon.
Hint: Some lines were ellipsized, use -l to show in full.

Status of OSD 11 before unit file update
...

[root@overcloud-osd-compute-0 ~]# journalctl | grep numa | grep starting
Dec 16 02:50:02 overcloud-osd-compute-0.localdomain numactl[83488]: starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
...

declare -A NUMASOCKET
while read TYPE SOCKET_NUM NIC ; do
    if [[ "$TYPE" == "NUMANode" ]]; then
	NUMASOCKET=$(echo $SOCKET_NUM | sed s/L//g);
    fi
    if [[ "$NIC" == "$OSD_NUMA_INTERFACE" ]]; then
	# because $NIC is the $OSD_NUMA_INTERFACE,
	# the NUMASOCKET has been set correctly above
	break # so stop looking
    fi
done < <(lstopo-no-graphics | tr -d [:punct:] | egrep "NUMANode|$OSD_NUMA_INTERFACE")

CMD=$(crudini --get $UNIT Service ExecStart)

crudini --verbose --set $UNIT Service ExecStart "$NUMA $CMD"

NUMA="/usr/bin/numactl -N $NUMASOCKET --preferred=$NUMASOCKET"

OSD_IDS=$(ls /var/lib/ceph/osd | awk 'BEGIN { FS = "-" } ; { print $2 }')
for OSD_ID in $OSD_IDS; do
  systemctl restart ceph-osd@$OSD_ID
done

There are many options passed in the command above. This subsection goes through each option in detail.

time openstack overcloud deploy --templates \

The above calls the openstack overcloud deploy command and uses the default location of the templates in /usr/share/openstack-tripleo-heat-templates/. The time command is used to time how long the deployment takes.

-r ~/custom-templates/custom-roles.yaml

The -r, or its longer extension --roles-file, overrides the default roles_data.yaml in the --templates directory. This is necessary because that file was copied, and the new OsdCompute role was created as described in Section 5.3, “Hyper Converged Role Definition”.

The next set of options passed is the following:

-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \

Passing --templates makes the deployment use the Heat templates in /usr/share/openstack-tripleo-heat-templates/ but the three environment files above, which reside in this directory, will not be used by the deployment by default. Thus, they need to be explicitly passed. Each of these Heat environment files perform the following functions:

The following includes the ~/custom-templates defined in Chapter 5, Define the Overcloud or in Chapter 6, Resource Isolation and Tuning

-e ~/custom-templates/network.yaml \
-e ~/custom-templates/ceph.yaml \
-e ~/custom-templates/compute.yaml \
-e ~/custom-templates/layout.yaml

The details of each environment file are covered in the following sections:

The order of the above arguments is necessary, since each environment file overrides the previous environment file.

Create a new JSON file describing the new nodes to be added. For example, if adding a server in a rack in slot U35, then a file like u35.json may contain the following:

 {
  "nodes": [
     {
         "pm_password": "PASSWORD",
         "name": "r730xd_u35",
         "pm_user": "root",
         "pm_addr": "10.19.136.28",
         "pm_type": "pxe_ipmitool",
         "mac": [
             "ec:f4:bb:ed:6f:e4"
         ],
         "arch": "x86_64",
	 "capabilities": "node:osd-compute-3,boot_option:local"
     }
  ]
}

 openstack baremetal import u35.json

For example, the server in U35 was assigned the ID 7250678a-a575-4159-840a-e7214e697165.

[stack@hci-director scale]$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| a94b75e3-369f-4b2d-b8cc-8ab272e23e89 | None | 629f3b1f-319f-4df7-8df1-0a9828f2f2f8 | power on    | active          | False       |
| 7ace7b2b-b549-414f-b83e-5f90299b4af3 | None | 4b354355-336d-44f2-9def-27c54cbcc4f5 | power on    | active          | False       |
| 8be1d83c-19cb-4605-b91d-928df163b513 | None | 29124fbb-ee1d-4322-a504-a1a190022f4e | power on    | active          | False       |
| e8411659-bc2b-4178-b66f-87098a1e6920 | None | 93199972-51ff-4405-979c-3c4aabdee7ce | power on    | active          | False       |
| 04679897-12e9-4637-9998-af8bee30b414 | None | e7578d80-0376-4df5-bbff-d4ac02eb1254 | power on    | active          | False       |
| 48b4987d-e778-48e1-ba74-88a08edf7719 | None | 586a5ef3-d530-47de-8ec0-8c98b30f880c | power on    | active          | False       |
| 7250678a-a575-4159-840a-e7214e697165 | None | None                                 | None        | available       | False       |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
[stack@hci-director scale]$

Maintenance mode prevents the server from being used for another purpose, e.g. another cloud operator adding an additional node at the same time.

 ironic node-set-maintenance 7250678a-a575-4159-840a-e7214e697165 true

 openstack baremetal introspection start 7250678a-a575-4159-840a-e7214e697165 true

The previous step takes time. The following command shows the status of the introspection.

[stack@hci-director ~]$  openstack baremetal introspection bulk status
+--------------------------------------+----------+-------+
| Node UUID                            | Finished | Error |
+--------------------------------------+----------+-------+
| a94b75e3-369f-4b2d-b8cc-8ab272e23e89 | True     | None  |
| 7ace7b2b-b549-414f-b83e-5f90299b4af3 | True     | None  |
| 8be1d83c-19cb-4605-b91d-928df163b513 | True     | None  |
| e8411659-bc2b-4178-b66f-87098a1e6920 | True     | None  |
| 04679897-12e9-4637-9998-af8bee30b414 | True     | None  |
| 48b4987d-e778-48e1-ba74-88a08edf7719 | True     | None  |
| 7250678a-a575-4159-840a-e7214e697165 | True     | None  |
+--------------------------------------+----------+-------+
[stack@hci-director ~]$

This step is necessary in order for the Red Hat OpenStack Platform director Nova Scheduler to select the new node when scaling the number of computes.

 ironic node-set-maintenance 7250678a-a575-4159-840a-e7214e697165 false

 openstack baremetal configure boot

The IDs of the kernel and ramdisk that were assigned to the new node are seen with the following command:

[stack@hci-director ~]$ ironic node-show 7250678a-a575-4159-840a-e7214e697165 | grep deploy_
| driver_info            | {u'deploy_kernel': u'e03c5677-2216-4120-95ad-b4354554a590',              |
|                        | u'ipmi_password': u'******', u'deploy_ramdisk': u'2c5957bd-              |
|                        | u'deploy_key': u'H3O1D1ETXCSSBDUMJY5YCCUFG12DJN0G', u'configdrive': u'H4 |
[stack@hci-director ~]$

The deploy_kernel and deploy_ramdisk are checked against what is in Glance. In the following example, the names bm-deploy-kernel and bm-deploy-ramdisk were assigned from the Glance database.

[stack@hci-director ~]$ openstack image list
+--------------------------------------+------------------------+--------+
| ID                                   | Name                   | Status |
+--------------------------------------+------------------------+--------+
| f7dce3db-3bbf-4670-8296-fa59492276c5 | bm-deploy-ramdisk      | active |
| 9b73446a-2c31-4672-a3e7-b189e105b2f9 | bm-deploy-kernel       | active |
| 653f9c4c-8afc-4320-b185-5eb1f5ecb7aa | overcloud-full         | active |
| 714b5f55-e64b-4968-a307-ff609cbcce6c | overcloud-full-initrd  | active |
| b9b62ec3-bfdb-43f7-887f-79fb79dcacc0 | overcloud-full-vmlinuz | active |
+--------------------------------------+------------------------+--------+
[stack@hci-director ~]$

Update ~/custom-templates/layout.yaml change the OsdComputeCount from 3 to 4 and add a new IP in each isolated network for the OsdCompute node. For example, change the following:

  OsdComputeIPs:
    internal_api:
      - 192.168.2.203
      - 192.168.2.204
      - 192.168.2.205
    tenant:
      - 192.168.3.203
      - 192.168.3.204
      - 192.168.3.205
    storage:
      - 172.16.1.203
      - 172.16.1.204
      - 172.16.1.205
    storage_mgmt:
      - 172.16.2.203
      - 172.16.2.204
      - 172.16.2.205

so that a .206 IP addresses is added as in the following:

  OsdComputeIPs:
    internal_api:
      - 192.168.2.203
      - 192.168.2.204
      - 192.168.2.205
      - 192.168.2.206
    tenant:
      - 192.168.3.203
      - 192.168.3.204
      - 192.168.3.205
      - 192.168.3.206
    storage:
      - 172.16.1.203
      - 172.16.1.204
      - 172.16.1.205
      - 172.16.1.206
    storage_mgmt:
      - 172.16.2.203
      - 172.16.2.204
      - 172.16.2.205
      - 172.16.2.206

See Section 5.5.3, “Configure scheduler hints to control node placement and IP assignment” for more information about the ~/custom-templates/layout.yaml file.

Use the same command that was used to deploy the overcloud to update the overcloud so that the changes made in the previous step are applied.

openstack overcloud deploy --templates \
-r ~/custom-templates/custom-roles.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e ~/custom-templates/network.yaml \
-e ~/custom-templates/ceph.yaml \
-e ~/custom-templates/layout.yaml

Use openstack server list to verify that the new OsdCompute node was added and is available. In the example below the new node, overcloud-osd-compute-3, is listed as ACTIVE.

[stack@hci-director ~]$ openstack server list
+--------------------------------------+-------------------------+--------+-----------------------+----------------+
| ID                                   | Name                    | Status | Networks              | Image Name     |
+--------------------------------------+-------------------------+--------+-----------------------+----------------+
| fc8686c1-a675-4c89-a508-cc1b34d5d220 | overcloud-controller-2  | ACTIVE | ctlplane=192.168.1.37 | overcloud-full |
| 7c6ae5f3-7e18-4aa2-a1f8-53145647a3de | overcloud-osd-compute-2 | ACTIVE | ctlplane=192.168.1.30 | overcloud-full |
| 851f76db-427c-42b3-8e0b-e8b4b19770f8 | overcloud-controller-0  | ACTIVE | ctlplane=192.168.1.33 | overcloud-full |
| e2906507-6a06-4c4d-bd15-9f7de455e91d | overcloud-controller-1  | ACTIVE | ctlplane=192.168.1.29 | overcloud-full |
| 0f93a712-b9eb-4f42-bc05-f2c8c2edfd81 | overcloud-osd-compute-0 | ACTIVE | ctlplane=192.168.1.32 | overcloud-full |
| 8f266c17-ff39-422e-a935-effb219c7782 | overcloud-osd-compute-1 | ACTIVE | ctlplane=192.168.1.24 | overcloud-full |
| 5fa641cf-b290-4a2a-b15e-494ab9d10d8a | overcloud-osd-compute-3 | ACTIVE | ctlplane=192.168.1.21 | overcloud-full |
+--------------------------------------+-------------------------+--------+-----------------------+----------------+
[stack@hci-director ~]$

The new Compute/Ceph Storage Node has been added the overcloud.

At this time of writing Red Hat OpenStack Platform director does not support the automated removal of a Red Hat Ceph Storage node, so steps in this section need to be done manually from one of the OpenStack Controller / Ceph Monitor nodes, unless otherwise indicated.

[root@overcloud-controller-0 ~]# ceph health
HEALTH_OK
[root@overcloud-controller-0 ~]#

HEALTH_WARN 1 nearfull osds
osd.2 is near full at 85%

In the example below, overcloud-osd-compute-3 will be removed, and the ceph osd tree command shows that its OSD numbers are 0 through 44 counting by fours.

[root@overcloud-controller-0 ~]# ceph osd tree
ID WEIGHT   TYPE NAME                        UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 52.37256 root default
-2 13.09314     host overcloud-osd-compute-3
 0  1.09109         osd.0                         up  1.00000          1.00000
 4  1.09109         osd.4                         up  1.00000          1.00000
 8  1.09109         osd.8                         up  1.00000          1.00000
12  1.09109         osd.12                        up  1.00000          1.00000
16  1.09109         osd.16                        up  1.00000          1.00000
20  1.09109         osd.20                        up  1.00000          1.00000
24  1.09109         osd.24                        up  1.00000          1.00000
28  1.09109         osd.28                        up  1.00000          1.00000
32  1.09109         osd.32                        up  1.00000          1.00000
36  1.09109         osd.36                        up  1.00000          1.00000
40  1.09109         osd.40                        up  1.00000          1.00000
44  1.09109         osd.44                        up  1.00000          1.00000
...

In a separate terminal, run the ceph -w command. This command is used to monitor the health of the Ceph cluster during OSD removal. The output of this command once started is similar to:

[root@overcloud-controller-0 ~]# ceph -w
    cluster eb2bb192-b1c9-11e6-9205-525400330666
     health HEALTH_OK
     monmap e2: 3 mons at {overcloud-controller-0=172.16.1.200:6789/0,overcloud-controller-1=172.16.1.201:6789/0,overcloud-controller-2=172.16.1.202:6789/0}
            election epoch 8, quorum 0,1,2 overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
     osdmap e139: 48 osds: 48 up, 48 in
            flags sortbitwise
      pgmap v106106: 1344 pgs, 6 pools, 11080 MB data, 4140 objects
            35416 MB used, 53594 GB / 53628 GB avail
                1344 active+clean

2016-11-29 02:13:17.058468 mon.0 [INF] pgmap v106106: 1344 pgs: 1344 active+clean; 11080 MB data, 35416 MB used, 53594 GB / 53628 GB avail
2016-11-29 02:15:03.674380 mon.0 [INF] pgmap v106107: 1344 pgs: 1344 active+clean; 11080 MB data, 35416 MB used, 53594 GB / 53628 GB avail
...

Use the ceph osd out <NUM> command to remove all twelve OSDs from the overcloud-osd-compute-3 node from the Ceph cluster. Allow for time between each OSD removal to ensure the cluster has time to complete the previous action before proceeding; this may be achieved by using a sleep statement. A script like the following, which uses seq to count from 0 to 44 by fours, may be used:

for i in $(seq 0 4 44); do
    ceph osd out $i;
    sleep 10;
done

Before running the above script, note the output of ceph osd stat with all OSDs up and in.

[root@overcloud-controller-0 ~]# ceph osd stat
     osdmap e173: 48 osds: 48 up, 48 in
            flags sortbitwise
[root@overcloud-controller-0 ~]#

The results of running the script above should look as follows:

[root@overcloud-controller-0 ~]# for i in $(seq 0 4 44); do ceph osd out $i; sleep 10; done
marked out osd.0.
marked out osd.4.
marked out osd.8.
marked out osd.12.
marked out osd.16.
marked out osd.20.
marked out osd.24.
marked out osd.28.
marked out osd.32.
marked out osd.36.
marked out osd.40.
marked out osd.44.
[root@overcloud-controller-0 ~]#

After the OSDs are marked as out, the output of the ceph osd stat command should show that twelve of the OSDs are no longer in but still up.

[root@overcloud-controller-0 ~]# ceph osd stat
     osdmap e217: 48 osds: 48 up, 36 in
            flags sortbitwise
[root@overcloud-controller-0 ~]#

The removal of the OSDs will cause Ceph to rebalance the cluster by migrating placement groups to other OSDs. The ceph -w command started in step 3 should show the placement group states as they change from active+clean to active, some degraded objects, and finally active+clean when migration completes.

An example of the output of ceph -w command started in step 3 as it changes looks like the following:

2016-11-29 02:16:06.372846 mon.2 [INF] from='client.? 172.16.1.200:0/1977099347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch
...
2016-11-29 02:16:07.624668 mon.0 [INF] osdmap e141: 48 osds: 48 up, 47 in
2016-11-29 02:16:07.714072 mon.0 [INF] pgmap v106111: 1344 pgs: 8 remapped+peering, 1336 active+clean; 11080 MB data, 34629 MB used, 52477 GB / 52511 GB avail
2016-11-29 02:16:07.624952 osd.46 [INF] 1.8e starting backfill to osd.2 from (0'0,0'0] MAX to 139'24162
2016-11-29 02:16:07.625000 osd.2 [INF] 1.ef starting backfill to osd.16 from (0'0,0'0] MAX to 139'17958
2016-11-29 02:16:07.625226 osd.46 [INF] 1.76 starting backfill to osd.25 from (0'0,0'0] MAX to 139'37918
2016-11-29 02:16:07.626074 osd.46 [INF] 1.8e starting backfill to osd.15 from (0'0,0'0] MAX to 139'24162
2016-11-29 02:16:07.626550 osd.21 [INF] 1.ff starting backfill to osd.46 from (0'0,0'0] MAX to 139'21304
2016-11-29 02:16:07.627698 osd.46 [INF] 1.32 starting backfill to osd.33 from (0'0,0'0] MAX to 139'24962
2016-11-29 02:16:08.682724 osd.45 [INF] 1.60 starting backfill to osd.16 from (0'0,0'0] MAX to 139'8346
2016-11-29 02:16:08.696306 mon.0 [INF] osdmap e142: 48 osds: 48 up, 47 in
2016-11-29 02:16:08.738872 mon.0 [INF] pgmap v106112: 1344 pgs: 6 peering, 9 remapped+peering, 1329 active+clean; 11080 MB data, 34629 MB used, 52477 GB / 52511 GB avail
2016-11-29 02:16:09.850909 mon.0 [INF] osdmap e143: 48 osds: 48 up, 47 in
...
2016-11-29 02:18:10.838365 mon.0 [INF] pgmap v106256: 1344 pgs: 7 activating, 1 active+recovering+degraded, 7 activating+degraded, 9 active+degraded, 70 peering, 1223 active+clean, 8 active+remapped, 19 remapped+peering; 11080 MB data, 33187 MB used, 40189 GB / 40221 GB avail; 167/12590 objects degraded (1.326%); 80/12590 objects misplaced (0.635%); 11031 kB/s, 249 objects/s recovering
...

Output like the above should continue as the Ceph cluster rebalances data, and eventually it returns to a health status of HEALTH_OK.

[root@overcloud-controller-0 ~]# ceph -s
    cluster eb2bb192-b1c9-11e6-9205-525400330666
     health HEALTH_OK
     monmap e2: 3 mons at {overcloud-controller-0=172.16.1.200:6789/0,overcloud-controller-1=172.16.1.201:6789/0,overcloud-controller-2=172.16.1.202:6789/0}
            election epoch 8, quorum 0,1,2 overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
     osdmap e217: 48 osds: 48 up, 36 in
            flags sortbitwise
      pgmap v106587: 1344 pgs, 6 pools, 11080 MB data, 4140 objects
            35093 MB used, 40187 GB / 40221 GB avail
                1344 active+clean
[root@overcloud-controller-0 ~]#

From the Red Hat OpenStack Platform director server, ssh into the node that is being removed and run systemctl stop ceph-osd.target to stop all OSDs.

Note how the output of ceph osd stat changes after systemctl command is run; the number of up OSDs changes from 48 to 36.

[root@overcloud-osd-compute-3 ~]# ceph osd stat
     osdmap e217: 48 osds: 48 up, 36 in
            flags sortbitwise
[root@overcloud-osd-compute-3 ~]# systemctl stop ceph-osd.target
[root@overcloud-osd-compute-3 ~]# ceph osd stat
     osdmap e218: 48 osds: 36 up, 36 in
            flags sortbitwise
[root@overcloud-osd-compute-3 ~]#

Be sure to run systemctl stop ceph-osd.target on the same node which hosts the OSDs, e.g. in this case, the OSDs from overcloud-osd-compute-3 will be removed, so the command is run on overcloud-osd-compute-3.

The script below does the following:

for i in $(seq 0 4 44); do
    ceph osd crush remove osd.$i
    sleep 10
    ceph auth del osd.$i
    sleep 10
    ceph osd rm $i
    sleep 10
done

Before removing the OSDs, note that they are in the CRUSH map for the Ceph storage node to be removed.

[root@overcloud-controller-0 ~]# ceph osd crush tree | grep overcloud-osd-compute-3 -A 20
                "name": "overcloud-osd-compute-3",
                "type": "host",
                "type_id": 1,
                "items": [
                    {
                        "id": 0,
                        "name": "osd.0",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    },
                    {
                        "id": 4,
                        "name": "osd.4",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    },
                    {
[root@overcloud-controller-0 ~]#

When the script above is executed, it looks like the following:

[root@overcloud-osd-compute-3 ~]# for i in $(seq 0 4 44); do
>     ceph osd crush remove osd.$i
>     sleep 10
>     ceph auth del osd.$i
>     sleep 10
>     ceph osd rm $i
>     sleep 10
> done
removed item id 0 name 'osd.0' from crush map
updated
removed osd.0
removed item id 4 name 'osd.4' from crush map
updated
removed osd.4
removed item id 8 name 'osd.8' from crush map
updated
removed osd.8
removed item id 12 name 'osd.12' from crush map
updated
removed osd.12
removed item id 16 name 'osd.16' from crush map
updated
removed osd.16
removed item id 20 name 'osd.20' from crush map
updated
removed osd.20
removed item id 24 name 'osd.24' from crush map
updated
removed osd.24
removed item id 28 name 'osd.28' from crush map
updated
removed osd.28
removed item id 32 name 'osd.32' from crush map
updated
removed osd.32
removed item id 36 name 'osd.36' from crush map
updated
removed osd.36
removed item id 40 name 'osd.40' from crush map
updated
removed osd.40
removed item id 44 name 'osd.44' from crush map
updated
removed osd.44
[root@overcloud-osd-compute-3 ~]#

The ceph osd stat command should now report that there are only 36 OSDs.

[root@overcloud-controller-0 ~]# ceph osd stat
     osdmap e300: 36 osds: 36 up, 36 in
            flags sortbitwise
[root@overcloud-controller-0 ~]#

When an OSD is removed from the CRUSH map, CRUSH recomputes which OSDs get the placement groups, and data re-balances accordingly. The CRUSH map may be checked after the OSDs are removed to verify that the update completed.

Observe that overcloud-osd-compute-3 has no OSDs:

[root@overcloud-controller-0 ~]# ceph osd crush tree | grep overcloud-osd-compute-3 -A 5
                "name": "overcloud-osd-compute-3",
                "type": "host",
                "type_id": 1,
                "items": []
            },
            {
[root@overcloud-controller-0 ~]#

Though the OSDs on overcloud-osd-compute-3 are no longer a member of the Ceph cluster, its Nova compute services are still functioning and will be removed in this subsection. The hardware will be shut off, and the Overlcoud Heat stack will no longer keep track of the node. All of the steps to do this should be carried out as the stack user on the Red Hat OpenStack Platform director system unless otherwise noted.

Before following this procedure, migrate any instances running on the compute node that will be removed to another compute node.

 source ~/overcloudrc

For example, overcloud-osd-compute-3 will be removed:

[stack@hci-director ~]$ nova service-list | grep compute-3
| 145 | nova-compute     | overcloud-osd-compute-3.localdomain | nova     | enabled | up    | 2016-11-29T03:40:32.000000 | -               |
[stack@hci-director ~]$

[stack@hci-director ~]$ nova service-disable overcloud-osd-compute-3.localdomain  nova-compute
+-------------------------------------+--------------+----------+
| Host                                | Binary       | Status   |
+-------------------------------------+--------------+----------+
| overcloud-osd-compute-3.localdomain | nova-compute | disabled |
+-------------------------------------+--------------+----------+
[stack@hci-director ~]$

 source ~/stackrc

[stack@hci-director ~]$ openstack server list | grep osd-compute-3
| 6b2a2e71-f9c8-4d5b-aaf8-dada97c90821 | overcloud-osd-compute-3 | ACTIVE | ctlplane=192.168.1.27 | overcloud-full |
[stack@hci-director ~]$

In the following example, the Nova ID is extracted with awk and egrep and set to the variable $nova_id

[stack@hci-director ~]$ nova_id=$(openstack server list | grep compute-3 | awk {'print $2'} | egrep -vi 'id|^$')
[stack@hci-director ~]$ echo $nova_id
6b2a2e71-f9c8-4d5b-aaf8-dada97c90821
[stack@hci-director ~]$

[stack@hci-director ~]$ time openstack overcloud node delete --stack overcloud $nova_id
deleting nodes [u'6b2a2e71-f9c8-4d5b-aaf8-dada97c90821'] from stack overcloud
Started Mistral Workflow. Execution ID: 396f123d-df5b-4f37-b137-83d33969b52b

real    1m50.662s
user    0m0.563s
sys     0m0.099s
[stack@hci-director ~]$

In the above example, the stack to delete the node from needs to be identified by name, "overcloud", instead of by its UUID. However, it will be possible to supply either the UUID or name after Red Hat Bugzilla 1399429 is resolved. It is no longer necessary when deleting a node to pass the Heat environment files with the -e option.

As shown by the time command output, the request to delete the node is accepted quickly. However, the Mistral workflow and Heat stack update will run in the background as it removes the compute node.

[stack@hci-director ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+----------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        | updated_time         |
+--------------------------------------+------------+--------------------+----------------------+----------------------+
| 23e7c364-7303-4af6-b54d-cfbf1b737680 | overcloud  | UPDATE_IN_PROGRESS | 2016-11-24T03:24:56Z | 2016-11-30T17:16:48Z |
+--------------------------------------+------------+--------------------+----------------------+----------------------+
[stack@hci-director ~]$

Confirm that Heat has finished updating the overcloud.

[stack@hci-director ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
| 23e7c364-7303-4af6-b54d-cfbf1b737680 | overcloud  | UPDATE_COMPLETE | 2016-11-24T03:24:56Z | 2016-11-30T17:16:48Z |
+--------------------------------------+------------+-----------------+----------------------+----------------------+
[stack@hci-director ~]$

In the example below, overcloud-osd-compute-3 is not included in the openstack server list output.

[stack@hci-director ~]$ openstack server list
+-------------------------+-------------------------+--------+-----------------------+----------------+
| ID                      | Name                    | Status | Networks              | Image Name     |
+-------------------------+-------------------------+--------+-----------------------+----------------+
| fc8686c1-a675-4c89-a508 | overcloud-controller-2  | ACTIVE | ctlplane=192.168.1.37 | overcloud-full |
| -cc1b34d5d220           |                         |        |                       |                |
| 7c6ae5f3-7e18-4aa2-a1f8 | overcloud-osd-compute-2 | ACTIVE | ctlplane=192.168.1.30 | overcloud-full |
| -53145647a3de           |                         |        |                       |                |
| 851f76db-427c-42b3      | overcloud-controller-0  | ACTIVE | ctlplane=192.168.1.33 | overcloud-full |
| -8e0b-e8b4b19770f8      |                         |        |                       |                |
| e2906507-6a06-4c4d-     | overcloud-controller-1  | ACTIVE | ctlplane=192.168.1.29 | overcloud-full |
| bd15-9f7de455e91d       |                         |        |                       |                |
| 0f93a712-b9eb-          | overcloud-osd-compute-0 | ACTIVE | ctlplane=192.168.1.32 | overcloud-full |
| 4f42-bc05-f2c8c2edfd81  |                         |        |                       |                |
| 8f266c17-ff39-422e-a935 | overcloud-osd-compute-1 | ACTIVE | ctlplane=192.168.1.24 | overcloud-full |
| -effb219c7782           |                         |        |                       |                |
+-------------------------+-------------------------+--------+-----------------------+----------------+
[stack@hci-director ~]$

[stack@hci-director ~]$ openstack baremetal node list
+-------------------+-------------+-------------------+-------------+--------------------+-------------+
| UUID              | Name        | Instance UUID     | Power State | Provisioning State | Maintenance |
+-------------------+-------------+-------------------+-------------+--------------------+-------------+
| c6498849-d8d8-404 | m630_slot13 | 851f76db-427c-    | power on    | active             | False       |
| 2-aa1c-           |             | 42b3-8e0b-        |             |                    |             |
| aa62ec2df17e      |             | e8b4b19770f8      |             |                    |             |
| a8b2e3b9-c62b-496 | m630_slot14 | e2906507-6a06     | power on    | active             | False       |
| 5-8a3d-           |             | -4c4d-            |             |                    |             |
| c4e7743ae78b      |             | bd15-9f7de455e91d |             |                    |             |
| f2d30a3a-8c74     | m630_slot15 | fc8686c1-a675-4c8 | power on    | active             | False       |
| -4fbf-afaa-       |             | 9-a508-cc1b34d5d2 |             |                    |             |
| fb666af55dfc      |             | 20                |             |                    |             |
| 8357d7b0-bd62-4b7 | r730xd_u29  | 0f93a712-b9eb-4f4 | power on    | active             | False       |
| 9-91f9-52c2a50985 |             | 2-bc05-f2c8c2edfd |             |                    |             |
| d9                |             | 81                |             |                    |             |
| fc6efdcb-ae5f-    | r730xd_u31  | 8f266c17-ff39-422 | power on    | active             | False       |
| 431d-             |             | e-a935-effb219c77 |             |                    |             |
| adf1-4dd034b4a0d3 |             | 82                |             |                    |             |
| 73d19120-6c93     | r730xd_u33  | 7c6ae5f3-7e18-4aa | power on    | active             | False       |
| -4f1b-ad1f-       |             | 2-a1f8-53145647a3 |             |                    |             |
| 4cce5913ba76      |             | de                |             |                    |             |
| a0b8b537-0975-406 | r730xd_u35  | None              | power off   | available          | False       |
| b-a346-e361464fd1 |             |                   |             |                    |             |
| e3                |             |                   |             |                    |             |
+-------------------+-------------+-------------------+-------------+--------------------+-------------+
[stack@hci-director ~]$

In the above, the server r730xd_u35 is powered off and available.

Authenticate back to the overcloud and observe the state of the nova-compute service offered by overcloud-osd-compute-3:

[stack@hci-director ~]$ source ~/overcloudrc
[stack@hci-director ~]$ nova service-list | grep osd-compute-3
| 145 | nova-compute     | overcloud-osd-compute-3.localdomain | nova     | disabled | down  | 2016-11-29T04:49:23.000000 | -               |
[stack@hci-director ~]$

In the above example, the overcloud has a nova-compute service on the overcloud-osd-compute-3 host, but it is currently marked as disabled and down.

Use nova service-delete 135 to remove the nova-compute service offered by overcloud-osd-compute-3.

The Compute/Ceph Storage Node has been fully removed.