Chapter 8. Configuring an OVS-DPDK deployment

Copy the usr/share/openstack-tripleo-heat-templates/plan-samples/plan-environment-derived-params.yaml file to a local directory and set the input parameters to suit your environment.

  workflow_parameters:
    tripleo.derive_params.v1.derive_parameters:
      # DPDK Parameters #
      # Specifies the minimum number of CPU physical cores to be allocated for DPDK
      # PMD threads. The actual allocation will be based on network config, if
      # the a DPDK port is associated with a numa node, then this configuration
      # will be used, else 1.
      num_phy_cores_per_numa_node_for_pmd: 1
      # Amount of memory to be configured as huge pages in percentage. Ouf the
      # total available memory (excluding the NovaReservedHostMemory), the
      # specified percentage of the remaining is configured as huge pages.
      huge_page_allocation_percentage: 50

Run the openstack overcloud deploy command with the following options:

Copy the derived parameters from the deploy command output to the network-environment.yaml file.

  # DPDK compute node.
  ComputeOvsDpdkParameters:
    KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on
    TunedProfileName: "cpu-partitioning"
    IsolCpusList: "1,2,3,4,5,6,7,9,10,17,18,19,20,21,22,23,11,12,13,14,15,25,26,27,28,29,30,31"
    NovaVcpuPinSet: ['2,3,4,5,6,7,18,19,20,21,22,23,10,11,12,13,14,15,26,27,28,29,30,31']
    NovaReservedHostMemory: 4096
    OvsDpdkSocketMemory: "1024,1024"
    OvsDpdkMemoryChannels: "4"
    OvsDpdkCoreList: "0,16,8,24"
    OvsPmdCoreList: "1,17,9,25"

Set the NeutronGlobalPhysnetMtu parameter in the network-environment.yaml file.

parameter_defaults:
  # MTU global configuration
  NeutronGlobalPhysnetMtu: 9000

Set the MTU value on the bridge to the Compute node in the controller.yaml file.

  -
    type: ovs_bridge
    name: br-link0
    use_dhcp: false
    members:
      -
        type: interface
        name: nic3
        mtu: 9000

Set the MTU values for an OVS-DPDK bond in the compute.yaml file:

- type: ovs_user_bridge
  name: br-link0
  use_dhcp: false
  members:
    - type: ovs_dpdk_bond
      name: dpdkbond0
      mtu: 9000
      rx_queue: 2
      members:
        - type: ovs_dpdk_port
          name: dpdk0
          mtu: 9000
          members:
            - type: interface
              name: nic4
        - type: ovs_dpdk_port
          name: dpdk1
          mtu: 9000
          members:
            - type: interface
              name: nic5

Ensure parameters for your DPDK compute role populate in network-environment.yaml. If necessary, copy these parameters from: derived OVS-DPDK.

 # DPDK compute node.
  ComputeOvsDpdkParameters:
    KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-7,17-23,9-15,25-31
    TunedProfileName: "cpu-partitioning"
    IsolCpusList: "1-7,17-23,9-15,25-31"
    NovaVcpuPinSet: ['2-7,18-23,10-15,26-31']
    NovaReservedHostMemory: 4096
    OvsDpdkSocketMemory: "1024,1024"
    OvsDpdkMemoryChannels: "4"
    OvsDpdkCoreList: "0,16,8,24"
    OvsPmdCoreList: "1,17,9,25"

Deploy the overcloud using the openstack overcloud deploy command.

Create an aggregate group and add relevant hosts for OVS-DPDK. Define metadata, for example dpdk=true, that matches defined flavor metadata.

 # openstack aggregate create dpdk_group
 # openstack aggregate add host dpdk_group [compute-host]
 # openstack aggregate set --property dpdk=true dpdk_group

Create a flavor.

# openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

Set additional flavor properties. Note that the defined metadata, dpdk=true, matches the defined metadata in the DPDK aggregate.

# openstack flavor set <flavor> --property dpdk=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB --property hw:emulator_threads_policy=isolate

In this example, m1.medium_huge_4cpu is the flavor name and the remaining parameters set the other properties for the flavor.

For details on the emulator threads policy for performance improvements, see: Configure Emulator Threads to run on a Dedicated Physical CPU.

Create the network.

# openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
# openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

Deploy an instance.

# openstack server create --flavor <flavor> --image <glance image> --nic net-id=<network ID>

Set the image properties.

# openstack image set --property hw_vif_multiqueue_enabled=true <image>

Set additional flavor properties.

# openstack flavor set --property hw:vif_multiqueue_enabled=true <flavor>

Review the bridge configuration and confirm that the bridge was created with the datapath_type=netdev.

# ovs-vsctl list bridge br0
_uuid               : bdce0825-e263-4d15-b256-f01222df96f3
auto_attach         : []
controller          : []
datapath_id         : "00002608cebd154d"
datapath_type       : netdev
datapath_version    : "<built-in>"
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : "br0"
netflow             : []
other_config        : {}
ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

Confirm that the docker container neutron_ovs_agent is configured to start automatically.

# docker inspect neutron_ovs_agent | grep -A1 RestartPolicy
            "RestartPolicy": {
                "Name": "always",

If the container has issues with starting, you can view any related messages.

# less /var/log/containers/neutron/openvswitch-agent.log

Confirm that the PMD CPU mask of the ovs-dpdk are pinned to the CPUs. In case of HT, use sibling CPUs.

For example, observe CPU 4:

# cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
4,20

Use CPU 4 and 20:

# ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

Display their status:

# tuna -t ovs-vswitchd -CP
thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
3219   OTHER 	0    	6     	1        	0   	handler24
3220   OTHER 	0    	6     	1        	0   	handler21
3221   OTHER 	0    	6     	1        	0   	handler22
3222   OTHER 	0    	6     	1        	0   	handler23
3223   OTHER 	0    	6     	1        	0   	handler25
3224   OTHER 	0    	6     	1        	0   	handler26
3225   OTHER 	0    	6     	1        	0   	handler27
3226   OTHER 	0    	6     	1        	0   	handler28
3227   OTHER 	0    	6     	2        	0   	handler31
3228   OTHER 	0    	6     	2        	4   	handler30
3229   OTHER 	0    	6     	2        	5   	handler32
3230   OTHER 	0    	6	953538      	431   revalidator29
3231   OTHER 	0    	6   1424258      	976   revalidator33
3232   OTHER 	0    	6   1424693      	836   revalidator34
3233   OTHER 	0    	6	951678      	503   revalidator36
3234   OTHER 	0    	6   1425128      	498   revalidator35
*3235   OTHER 	0    	4	151123       	51       	pmd37*
*3236   OTHER 	0   	20	298967       	48       	pmd38*
3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
3165   OTHER 	0    	6	237634        	0   vhost_thread1
3166   OTHER 	0    	6  	3665        	0       	urcu2