Chapter 9. Configuring a heterogeneous compute cluster

This section describes how to deploy an SR-IOV Compute node and an OVS-DPDK Compute node in the same environment. This deployment uses custom roles for OVS-DPDK and SR-IOV with role-specific parameters defined in the network-environment.yaml file. The process to create and deploy a composable role includes:

  • Define the SR-IOV and OVS-DPDK custom roles in a local copy of the roles_data.yaml file.
  • Set the role-specific parameters for the SR-IOV role and the OVS-DPDK role in the network_environment.yaml file.
  • Deploy the overcloud with this updated set of roles.

You must install and configure the undercloud before you can deploy a heterogeneous compute cluster in the overcloud. See the Director Installation and Usage Guide for details.

Note

Ensure that you create OpenStack flavors that match these custom roles.

Important

You must determine the best values for the OVS-DPDK parameters that you set in the network-environment.yaml file to optimize your OpenStack network for OVS-DPDK. See Section 8.1, “Deriving DPDK parameters with workflows” for details.

9.1. Naming conventions

We recommend that you follow a consistent naming convention when you use custom roles in your OpenStack deployment, especially with a heterogeneous compute cluster. This naming convention can assist you when creating the following files and configurations:

  • instackenv.json - To differentiate between the compute node that uses SR-IOV interfaces from the compute node that uses DPDK interfaces.

    "name":"computeovsdpdk-0"
  • roles_data.yaml - To differentiate between compute-based roles that support SR-IOV from compute-based roles that support DPDK.

    `ComputeOvsDpdk`
  • network_environment.yaml -

    • To ensure that you match the custom role to the correct flavor name.

       `OvercloudComputeOvsDpdkFlavor: computeovsdpdk`
    • To include the correct custom role name for any role-specific parameters.

      `ComputeOvsDpdkParameters`
  • nic-config file names - To differentiate NIC yaml files for compute nodes that support SR-IOV from compute nodes that support DPDK interfaces.
  • Flavor creation - To help you match a flavor and capabilities:profile value to the appropriate bare metal node and custom role.

    # openstack flavor create --id auto --ram 4096 --disk 40 --vcpus 4 computeovsdpdk
    # openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="computeovsdpdk" computeovsdpdk
  • Bare metal node - To ensure that you match the bare metal node with the appropriate hardware and capability:profile value.

     # openstack baremetal node update computeovsdpdk-0 add properties/capabilities='profile:computeovsdpdk,boot_option:local'
Note

The flavor name does not have to match the capabilities:profile value for the flavor, but the flavor capabilities:profile value must match the bare metal node properties/capabilities='profile value. All three use computeovsdpdk in this example.

Note

Ensure that all your nodes used for a custom role and profile have the same CPU, RAM, and PCI hardware topology.

In this example, the ComputeOvsDpdk and ComputeSriov are custom roles for compute nodes to enable DPDK or SR-IOV only on the nodes that have the appropriate NICs. The existing set of default roles provided by Red Hat OpenStack Platform is stored in the /home/stack/roles_data.yaml file.

9.2. Creating the SR-IOV and OVS-DPDK custom roles

Red Hat OpenStack provides a set of default roles in the roles_data.yaml file. You can create your own roles_data.yaml file to support the roles you need.

To create the custom roles to support SR-IOV and OVS-DPDK:

  1. Create the ComputeSriov.yaml file in a local directory and add the definition of this role:

    ###############################################################################
    # Role: ComputeSriov                                                          #
    ###############################################################################
    - name: ComputeSriov
      description: |
        Compute SR-IOV role
      CountDefault: 1
      networks:
        - InternalApi
        - Tenant
        - Storage
      HostnameFormatDefault: 'computesriov-%index%'
      disable_upgrade_deployment: True
      ServicesDefault:
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CephClient
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::ComputeCeilometerAgent
        - OS::TripleO::Services::ComputeNeutronCorePlugin
        - OS::TripleO::Services::ComputeNeutronL3Agent
        - OS::TripleO::Services::ComputeNeutronMetadataAgent
        - OS::TripleO::Services::ComputeNeutronOvsAgent
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::FluentdClient
        - OS::TripleO::Services::Iscsid
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::MySQLClient
        - OS::TripleO::Services::NeutronLinuxbridgeAgent
        - OS::TripleO::Services::NeutronSriovAgent
        - OS::TripleO::Services::NeutronVppAgent
        - OS::TripleO::Services::NovaCompute
        - OS::TripleO::Services::NovaLibvirt
        - OS::TripleO::Services::NovaMigrationTarget
        - OS::TripleO::Services::Ntp
        - OS::TripleO::Services::OpenDaylightOvs
        - OS::TripleO::Services::Securetty
        - OS::TripleO::Services::SensuClient
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Sshd
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
        - OS::TripleO::Services::Tuned
        - OS::TripleO::Services::Vpp
        - OS::TripleO::Services::OVNController
  2. Create the ComputeOvsDpdk.yaml file in a local directory and add the definition of this role:

    ###############################################################################
    # Role: ComputeOvsDpdk                                                        #
    ###############################################################################
    - name: ComputeOvsDpdk
      description: |
        Compute OVS DPDK Role
      CountDefault: 1
      networks:
        - InternalApi
        - Tenant
        - Storage
      HostnameFormatDefault: 'computeovsdpdk-%index%'
      disable_upgrade_deployment: True
      ServicesDefault:
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CephClient
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::ComputeCeilometerAgent
        - OS::TripleO::Services::ComputeNeutronCorePlugin
        - OS::TripleO::Services::ComputeNeutronL3Agent
        - OS::TripleO::Services::ComputeNeutronMetadataAgent
        - OS::TripleO::Services::ComputeNeutronOvsDpdk
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::FluentdClient
        - OS::TripleO::Services::Iscsid
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::MySQLClient
        - OS::TripleO::Services::NovaCompute
        - OS::TripleO::Services::NovaLibvirt
        - OS::TripleO::Services::Ntp
        - OS::TripleO::Services::OpenDaylightOvs
        - OS::TripleO::Services::Securetty
        - OS::TripleO::Services::SensuClient
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Sshd
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
  3. Generate roles_data.yaml for the ComputeSriov and ComputeOvsDpdk roles and any other roles you need for your deployment:

    # openstack overcloud roles generate --roles-path templates/openstack-tripleo-heat-templates/roles -o roles_data.yaml Controller ComputeSriov ComputeOvsDpdk

9.3. Defining the SR-IOV and OVS-DPDK role-specific parameters

Important

You must determine the best values for the OVS-DPDK parameters that you set in the network-environment.yaml file to optimize your OpenStack network for OVS-DPDK. See Planning your OVS-DPDK Deployment and Section 8.1, “Deriving DPDK parameters with workflows” for details.

To configure SR-IOV and OVS-DPDK role-specific parameters in the network-environment.yaml file:

  1. Add the resource mapping for the OVS-DPDK and SR-IOV services to the network-environment.yaml file along with the network configuration for these nodes:

      resource_registry:
        # Specify the relative/absolute path to the config files you want to use for override the default.
        OS::TripleO::ComputeSriov::Net::SoftwareConfig: nic-configs/compute-sriov.yaml
        OS::TripleO::ComputeOvsDpdk::Net::SoftwareConfig: nic-configs/compute-ovs-dpdk.yaml
        OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml
  2. Specify the flavors for each role:

    OvercloudControlFlavor: controller
    OvercloudComputeOvsDpdkFlavor: computeovsdpdk
    OvercloudComputeSriovFlavor: computesriov
  3. Specify the number of nodes to deploy for each role:

      #Number of nodes to deploy.
      ControllerCount: 1
      ComputeOvsDpdkCount: 1
      ComputeSriovCount: 1
  4. Configure the SR-IOV parameters:

    1. Configure the Compute pci_passthrough_whitelist parameter, and set devname for the SR-IOV interface. The whitelist sets the PCI devices available to instances.

        NovaPCIPassthrough:
          - devname: "p7p1"
            physical_network: "tenant"
          - devname: "p7p2"
            physical_network: "tenant"
    2. Specify the physical network and SR-IOV interface in the format - PHYSICAL_NETWORK:PHYSICAL DEVICE:

      All physical networks listed in the network_vlan_ranges on the server should have mappings to the appropriate interfaces on each agent.

        NeutronPhysicalDevMappings: "tenant:p7p1,tenant:p7p2"

      This example uses tenant as the physical_network name.

    3. Provide the number of Virtual Functions (VFs) to be reserved for each SR-IOV interface:

        NeutronSriovNumVFs: "p7p1:5,p7p2:5"

      This example reserves 5 VFs for the SR-IOV interface.

      Note

      Red Hat OpenStack Platform supports the number of VFs supported by the NIC vendor. See Deployment Limits for Red Hat OpenStack Platform for other related details.

  5. Configure the role-specific parameters for SR-IOV:

      # SR-IOV compute node.
      ComputeSriovParameters:
        KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
        TunedProfileName: "cpu-partitioning"
        IsolCpusList: "2-19,22-39"
        NovaVcpuPinSet: ['4-19,24-39']
        NovaReservedHostMemory: 4096
  6. Configure the role-specific parameters for OVS-DPDK:

      # DPDK compute node.
      ComputeOvsDpdkParameters:
        KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
        TunedProfileName: "cpu-partitioning"
        IsolCpusList: "2-19,22-39"
        NovaVcpuPinSet: ['4-19,24-39']
        NovaReservedHostMemory: 4096
        OvsDpdkSocketMemory: "3072,1024"
        OvsDpdkMemoryChannels: "4"
        OvsDpdkCoreList: "0,20,1,21"
        OvsPmdCoreList: "2,22,3,23"
    Note

    You must assign at least one CPU (with sibling thread) on each NUMA node with or without DPDK NICs present for DPDK PMD to avoid failures in creating guest instances.

  7. Configure the remainder of the network-environment.yaml file to override the default parameters from the neutron-ovs-dpdk-agent.yaml and neutron-sriov-agent.yaml files as needed for your OpenStack deployment.

9.4. Configuring SR-IOV and DPDK compute nodes

To support an SR-IOV Compute node and an OVS-DPDK Compute node:

  1. Create the compute-sriov.yaml file to support SR-IOV interfaces.

    1. Create the control plane Linux bond for an isolated network:

        - type: linux_bond
          name: bond_api
          bonding_options: "mode=active-backup"
          use_dhcp: false
          dns_servers:
            get_param: DnsServers
          members:
          - type: interface
            name: nic3
            primary: true
          - type: interface
            name: nic4
    2. Assign VLANs to this Linux bond:

        - type: vlan
          vlan_id:
            get_param: InternalApiNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: InternalApiIpSubnet
      
        - type: vlan
          vlan_id:
            get_param: TenantNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: TenantIpSubnet
      
        - type: vlan
          vlan_id:
            get_param: StorageNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: StorageIpSubnet
    3. Create the SR-IOV interfaces to the Controller:

        - type: interface
          name: p7p1
          mtu: 9000
          use_dhcp: false
          defroute: false
          nm_controlled: true
          hotplug: true
      
        - type: interface
          name: p7p2
          mtu: 9000
          use_dhcp: false
          defroute: false
          nm_controlled: true
          hotplug: true
  2. Create the compute-ovsdpdk.yaml file to support DPDK interfaces.

    1. Create the control plane Linux bond for an isolated network:

        - type: linux_bond
          name: bond_api
          bonding_options: "mode=active-backup"
          use_dhcp: false
          dns_servers:
            get_param: DnsServers
          members:
          - type: interface
            name: nic3
            primary: true
          - type: interface
            name: nic4
    2. Assign VLANs to this Linux bond:

        - type: vlan
          vlan_id:
            get_param: InternalApiNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: InternalApiIpSubnet
      
        - type: vlan
          vlan_id:
            get_param: TenantNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: TenantIpSubnet
      
        - type: vlan
          vlan_id:
            get_param: StorageNetworkVlanID
          device: bond_api
          addresses:
          - ip_netmask:
              get_param: StorageIpSubnet
    3. Set the bridge with an OVS-DPDK interface to link to the Controller:

        - type: ovs_user_bridge
          name: br-link0
          use_dhcp: false
          members:
          - type: ovs_dpdk_port
            name: dpdk0
            mtu: 9000
            members:
            - type: interface
              name: nic7
      Note

      To include multiple DPDK devices, repeat the type code section for each DPDK device you want to add.

    Note

    When using OVS-DPDK, all bridges on the same Compute node should be of type ovs_user_bridge. The director may accept the configuration, but Red Hat OpenStack Platform does not support mixing ovs_bridge and ovs_user_bridge on the same node.

9.5. Deploying the overcloud

The following example defines the openstack overcloud deploy Bash script that uses composable roles:

#!/bin/bash

openstack overcloud deploy \
--templates \
-r /home/stack/ospd-12-sriov-dpdk-heterogeneous-cluster/roles_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-sriov.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \
-e /home/stack/ospd-12-vlan-sriov-two-ports-ctlplane-bonding/docker-images.yaml \
-e /home/stack/ospd-12-sriov-dpdk-heterogeneous-cluster/network-environment.yaml

Where:

  • /home/stack/ospd-12-sriov-dpdk-heterogeneous-cluster/roles_data.yaml is the location of the updated roles_data.yaml file, which defines the OVS-DPDK and the SR-IOV composable roles.
  • /home/stack/ospd-12-sriov-dpdk-heterogeneous-cluster/network-environment.yaml contains the role-specific parameters for the SR-IOV Compute node and the OVS-DPDK Compute node.