Chapter 10. Example: Configuring OVS-DPDK and SR-IOV with VXLAN tunnelling

This section describes how to deploy Compute nodes with both OVS-DPDK and SR-IOV interfaces. The cluster installs with ML2/OVS and VXLAN tunnelling.

Important

You must determine the best values for the OVS-DPDK parameters that you set in the network-environment.yaml file to optimize your OpenStack network for OVS-DPDK. See Deriving DPDK parameters with workflows for details.

10.1. Configuring roles data

Red Hat OpenStack Platform provides a set of default roles in the roles_data.yaml file. You can create your own roles_data.yaml file to support the roles you require.

In this example, the ComputeOvsDpdkSriov role is created. For information on creating roles in Red Hat OpenStack Platform, see Advanced Overcloud Customization. For details on the specific role used for this example, see roles_data.yaml.

10.2. Configuring OVS-DPDK parameters

Important

You must determine the best values for the OVS-DPDK parameters that you set in the network-environment.yaml file to optimize your OpenStack network for OVS-DPDK. For details, see: Deriving DPDK parameters with workflows.

  1. Add the custom resources for OVS-DPDK under resource_registry: 

      resource_registry:
    # Specify the relative/absolute path to the config files you want to use for override the default.
    OS::TripleO::ComputeOvsDpdkSriov::Net::SoftwareConfig: nic-configs/computeovsdpdksriov.yaml
    OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml

  2. Under parameter_defaults, set the tunnel type to vxlan and the network type to vxlan,vlan: 

    NeutronTunnelTypes: 'vxlan'
NeutronNetworkType: 'vxlan,vlan'

  3. Under parameters_defaults, set the bridge mapping: 

    # The OVS logical->physical bridge mappings to use.
NeutronBridgeMappings:
  - dpdk-mgmt:br-link0

  4. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role: 

      ##########################
  # OVS DPDK configuration #
  ##########################
  ComputeOvsDpdkSriovParameters:
    KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
    TunedProfileName: "cpu-partitioning"
    IsolCpusList: "2-19,22-39"
    NovaVcpuPinSet: ['4-19,24-39']
    NovaReservedHostMemory: 4096
    OvsDpdkSocketMemory: "3072,1024"
    OvsDpdkMemoryChannels: "4"
    OvsDpdkCoreList: "0,20,1,21"
    OvsPmdCoreList: "2,22,3,23"
    NovaComputeCpuSharedSet: [0,20,1,21]
    NovaLibvirtRxQueueSize: 1024
    NovaLibvirtTxQueueSize: 1024
    Note

    You must assign at least one CPU with sibling thread on each NUMA node with or without DPDK NICs present for DPDK PMD to avoid failures in creating guest instances. This is expressed in the example through the OvsPmdCoreList parameter, having cores 2 and 22 from NUMA 1, and having cores 3 and 23 from NUMA 2.

    Note

    These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the VMs is the boot parameter minus the OvsDpdkSocketMemory.

    You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

    Note

    OvsDPDKCoreList and OvsDpdkMemoryChannels are the required settings for this procedure. If you deploy DPDK without appropriate values, the deployment might fail or be unstable.

  5. Configure the role-specific parameters for SR-IOV: 

      NovaPCIPassthrough:
    - devname: "p7p3"
      trusted: "true"
      physical_network: "sriov-1"
    - devname: "p7p4"
      trusted: "true"
      physical_network: "sriov-2"

10.3. Configuring the Controller node

  1. Create the control plane Linux bond for an isolated network. 

      - type: linux_bond
    name: bond_api
    bonding_options: "mode=active-backup"
    use_dhcp: false
    dns_servers:
      get_param: DnsServers
    members:
    - type: interface
      name: nic2
      primary: true

  2. Assign VLANs to this Linux bond. 

      - type: vlan
    vlan_id:
      get_param: InternalApiNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: InternalApiIpSubnet

  - type: vlan
    vlan_id:
      get_param: StorageNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: StorageIpSubnet

  - type: vlan
    vlan_id:
      get_param: StorageMgmtNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: StorageMgmtIpSubnet

  - type: vlan
    vlan_id:
      get_param: ExternalNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: ExternalIpSubnet
    routes:
    - default: true
      next_hop:
        get_param: ExternalInterfaceDefaultRoute

  3. Create the OVS bridge for access to neutron-dhcp-agent and neutron-metadata-agent services. 

      - type: ovs_bridge
    name: br-link0
    use_dhcp: false
    mtu: 9000
    members:
    - type: interface
      name: nic3
      mtu: 9000
    - type: vlan
      vlan_id:
        get_param: TenantNetworkVlanID
      mtu: 9000
      addresses:
      - ip_netmask:
          get_param: TenantIpSubnet

10.4. Configuring the Compute node for DPDK and SR-IOV

Create the computeovsdpdksriov.yaml file from the default compute.yaml file and make the following changes:

  1. Create the control plane Linux bond for an isolated network. 

      - type: linux_bond
    name: bond_api
    bonding_options: "mode=active-backup"
    use_dhcp: false
    dns_servers:
      get_param: DnsServers
    members:
    - type: interface
      name: nic3
      primary: true
    - type: interface
      name: nic4

  2. Assign VLANs to this Linux bond. 

      - type: vlan
    vlan_id:
      get_param: InternalApiNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: InternalApiIpSubnet

  - type: vlan
    vlan_id:
      get_param: StorageNetworkVlanID
    device: bond_api
    addresses:
    - ip_netmask:
        get_param: StorageIpSubnet

  3. Set a bridge with a DPDK port to link to the controller. 

      - type: ovs_user_bridge
    name: br-link0
    use_dhcp: false
    ovs_extra:
      - str_replace:
          template: set port br-link0 tag=_VLAN_TAG_
          params:
            _VLAN_TAG_:
               get_param: TenantNetworkVlanID
    addresses:
      - ip_netmask:
          get_param: TenantIpSubnet
    members:
      - type: ovs_dpdk_bond
        name: dpdkbond0
        mtu: 9000
        rx_queue: 2
        members:
          - type: ovs_dpdk_port
            name: dpdk0
            members:
              - type: interface
                name: nic7
          - type: ovs_dpdk_port
            name: dpdk1
            members:
              - type: interface
                name: nic8
    Note

    To include multiple DPDK devices, repeat the type code section for each DPDK device that you want to add.

    Note

    When using OVS-DPDK, all bridges on the same Compute node must be of type ovs_user_bridge. The director might accept the configuration, but Red Hat OpenStack Platform does not support mixing ovs_bridge and ovs_user_bridge on the same node.

10.5. Deploying the overcloud

  1. Run the overcloud_deploy.sh script to deploy the overcloud. 
openstack overcloud deploy \
--templates \
-r /home/stack/ospd-15-vxlan-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/roles_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-ovs-dpdk.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/neutron-sriov.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/collectd-environment.yaml \
-e /home/stack/ospd-15-vxlan-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/api-policies.yaml \
-e /home/stack/ospd-15-vxlan-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/network-environment.yaml \
-e /home/stack/overcloud_images.yaml \
--log-file overcloud_install.log &> overcloud_install.log