Chapter 8. NFV performance tuning with OVS-DPDK

OVS-DPDK is based on polling the Rx (Poll Mode Driver) queues both on the OVS-DPDK vSwitch that resides in user space of the host as well as the PMD used in the guest VM (VNF). Polling activity is directly mapped to CPU cores. If we we attempt to use OVS-DPDK without any tuning, a random CPU (probably first in the list - core 0) may get picked for all the activities:

  • Normal non-DPDK related processes that require CPU
  • PMD threads that need CPU for polling on host vSwitch (OVS-DPDK)
  • VM (VNFs) where PMD runs as a part of DPDK in the guest

To maximize throughput, enable tuning with Red Hat OpenStack director to reserve CPU cores for each OVS-DPDK activity.

8.1. Hardware requirements

Achieving zero loss also requires:

Additionally the BIOS settings shown in Table 4 need to be set as shown.

SettingValue

C3 Power State

Disabled

C6 Power State

Disabled

MLC Steamer

Enabled

MLC Spacial Prefetcher

Enabled

DCU Data Prefetcher

Enabled

DCA

Enabled

CPU Power and Performance

Performance

Memory RAS and Performance Config → NUMA Optimized

Enabled

Table 4: BIOS settings for OVS-DPDK

8.2. Setting OOO Heat templates based on your hardware

After configuring the BIOS settings as described in the hardware section, in order for the tuning to be performed, the following parameters that will be consumed by OOO heat templates as a part of OpenStack overcloud deployment by Red Hat Openstack director need to be configured in the network-environment.yaml file.

HostCpusList - List of all CPUs (cores) that are not going to be used for OVS-DPDK. Parameter set in network-environment.yaml file:

# Set a list or range of physical CPU cores that should be excluded from tuning:
HostCpusList: "'0,1,2,3,4,5,6,7,8,9,10,11,24,25,26,27,28,29,30,31,32,33,34,35'"
Note

It is a good practice to exclude siblings of CPU cores 0 through 11, in this case 24 through 35 from being used for DPDK. This can be observed in virsh capabilities output.

HostIsolatedCoreList - List of CPUs used for tuning
# HostIsolatedCoreList = NeutronDpdkCoreList(12-15) + NovaVcpuPinSet(16-23)
  HostIsolatedCoreList: "12-23"
Note

HostIsolatedCoreList and HostCpusList are mutually exclusive. In the above example, CPUs 0-11 and 24-35 are reserved for non-DPDK functions.

NeutronDpdkCoreList - List of cpus to be given for pmd-cpu-mask
  # List of cores to be used for DPDK Poll Mode Driver
  NeutronDpdkCoreList: "'12,13,14,15'"

NovaVcpuPinSet - List of cpus to be used for guest vms (used by nova)

 NovaVcpuPinSet: ['16-23']
Note

NeutronDpdkCoreList and NovaVcpuPinSet should be selected from NUMA node 1 in this case because the DPDK NICs are mapped to NUMA node 1

Arguments passed on to the kernel.

  ComputeKernelArgs: "iommu=pt intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=12 isolcpus=12-23"

8.2.1. Examining the hardware for tuning

In the NFV validation lab, the last 2 NICs (NICs 5 & 6 or ens6f0 and ens6f1) are used for dataplane. Let’s start by seeing which NUMA node(s) the NICs are mapped to:

8.2.1.1. Mapping DPDK NIC cards to NUMA nodes

To determine which NUMA node the NICs are mapped to, use the following command:

[root@se-nfv-srv12 ~]# cat /sys/class/net/ens6f0/device/numa_node
1
[root@se-nfv-srv12 ~]# cat /sys/class/net/ens6f1/device/numa_node
1

It can be seen from the above output that both ens6f0 and ens6f1 are mapped to NUMA node 1.

Next thing to do is to get the CPU cores mapped to socket 1 (NUMA node 1).

8.2.1.2. CPU (Processor) to Socket Mapping

CPU to socket mapping can be gleaned using the commands shown below on one of the compute nodes. While selecting CPUs to be assigned to PMD threads it is important to keep in mind the CPU cores associated with the sibling threads. Avoid assigning cores associated to sibling threads to VCpuPinSet for use by VMs (VNFs). Mapping of CPU cores to sockets and sibling information can be obtained using one of the following two methods:

  • Method A: Using “virsh capabilities” command
  • Method B: Using native linux commands
Note

Typically, all this information would need to be gathered on one of the compute nodes (or any server that has hardware that is identical to the compute nodes) before Red Hat OpenStack director based install. Both methods A and B are covered below just in case virsh capabilities is not available at the time of gathering the required information.

8.2.1.2.1. Method A (Using virsh capabilities):

Run the command virsh capabilities on a compute node or a host that has hardware and configuration that is identical to that of a compute node:

[root@overcloud-compute-0 ~]# virsh capabilities
<capabilities>

  <host>
    <uuid>b1c41b1d-e33c-e511-a9fb-2c600ccd092c</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Broadwell</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='12' threads='2'/>
      <feature name='vme'/>
      < == SNIP === >
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>

 <topology>
      <cells num='2'>
        <cell id='0'>
          <cpus num='24'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,24'/>
</distances>
          <cpus num='24'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,24'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,25'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,26'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,27'/>
            <cpu id='4' socket_id='0' core_id='4' siblings='4,28'/>
            <cpu id='5' socket_id='0' core_id='5' siblings='5,29'/>
            <cpu id='6' socket_id='0' core_id='8' siblings='6,30'/>
            <cpu id='7' socket_id='0' core_id='9' siblings='7,31'/>
            <cpu id='8' socket_id='0' core_id='10' siblings='8,32'/>
            <cpu id='9' socket_id='0' core_id='11' siblings='9,33'/>
            <cpu id='10' socket_id='0' core_id='12' siblings='10,34'/>
            <cpu id='11' socket_id='0' core_id='13' siblings='11,35'/>
            < ==== SNIP ==== >
        </cell>

  <cpus num='24'>
            <cpu id='12' socket_id='1' core_id='0' siblings='12,36'/>
            <cpu id='13' socket_id='1' core_id='1' siblings='13,37'/>
            <cpu id='14' socket_id='1' core_id='2' siblings='14,38'/>
            <cpu id='15' socket_id='1' core_id='3' siblings='15,39'/>
            <cpu id='16' socket_id='1' core_id='4' siblings='16,40'/>
            <cpu id='17' socket_id='1' core_id='5' siblings='17,41'/>
            <cpu id='18' socket_id='1' core_id='8' siblings='18,42'/>
            <cpu id='19' socket_id='1' core_id='9' siblings='19,43'/>
            <cpu id='20' socket_id='1' core_id='10' siblings='20,44'/>
            <cpu id='21' socket_id='1' core_id='11' siblings='21,45'/>
            <cpu id='22' socket_id='1' core_id='12' siblings='22,46'/>
            <cpu id='23' socket_id='1' core_id='13' siblings='23,47'/>
            <cpu id='36' socket_id='1' core_id='0' siblings='12,36'/>
            <cpu id='37' socket_id='1' core_id='1' siblings='13,37'/>
            <cpu id='38' socket_id='1' core_id='2' siblings='14,38'/>
            <cpu id='39' socket_id='1' core_id='3' siblings='15,39'/>
            <cpu id='40' socket_id='1' core_id='4' siblings='16,40'/>
            <cpu id='41' socket_id='1' core_id='5' siblings='17,41'/>
            <cpu id='42' socket_id='1' core_id='8' siblings='18,42'/>
            <cpu id='43' socket_id='1' core_id='9' siblings='19,43'/>
            <cpu id='44' socket_id='1' core_id='10' siblings='20,44'/>
            <cpu id='45' socket_id='1' core_id='11' siblings='21,45'/>
            <cpu id='46' socket_id='1' core_id='12' siblings='22,46'/>
            <cpu id='47' socket_id='1' core_id='13' siblings='23,47'/>
          </cpus>
        </cell>
      </cells>
    </topology>
Note

From the CPU section of the above output, <topology sockets='1' cores='12' threads='2'/>. This implies Hyper-threading enabled threads = 2

Note

From above output, siblings 0-11 and 24-35 are used for HostCpusList, cores 12-15 are used for NeutronDpdkCoreList for PMD threads and lastly, cores 16-23 are used to assign NovaVcpuPinSet

8.2.1.2.2. Method B (Using native Linux commands)

Checking if hyper-threading has been enabled:

[root@overcloud-compute-0 ~]# dmidecode -t processor | grep HTT
                HTT (Multi-threading)
                HTT (Multi-threading)
Note

Even though the hosts (compute nodes) used in the NFV validation lab have hyperthreading enabled on them, the setup and tests shown here are for single-queue. In this setup, one CPU core is required for polling each physical NIC(PNIC). Since there are 2 DPDK ports, 2 CPUs are consumed for polling PNICs. Additionally, one CPU core is required to service each vHost user port (that connects the VM to the PNIC). If we have 2 VMs with one port each that accounts for 2 additional CPU cores. Based on the above configuration, 2 DPDK PNICs and 2 VMs with 1 port each can be accommodated.

Obtain the physical socket (NUMA node) to CPU core mapping:

[root@se-nfv-srv12 ~]#  egrep -e "processor.*:" -e ^physical /proc/cpuinfo|xargs -l2 echo | awk '{print "socket " $7 "\t" "core " $3}' | sort -n -k2 -k4
socket 0        core 0
socket 0        core 1
socket 0        core 2
socket 0        core 3
socket 0        core 4
socket 0        core 5
socket 0        core 6
socket 0        core 7
socket 0        core 8
socket 0        core 9
socket 0        core 10
socket 0        core 11
socket 0        core 24
socket 0        core 25
socket 0        core 26
socket 0        core 27
socket 0        core 28
socket 0        core 29
socket 0        core 30
socket 0        core 31
socket 0        core 32
socket 0        core 33
socket 0        core 34
socket 0        core 35
socket 1        core 12
socket 1        core 13
socket 1        core 14
socket 1        core 15
socket 1        core 16
socket 1        core 17
socket 1        core 18
socket 1        core 19
socket 1        core 20
socket 1        core 21
socket 1        core 22
socket 1        core 23
socket 1        core 36
socket 1        core 37
socket 1        core 38
socket 1        core 39
socket 1        core 40
socket 1        core 41
socket 1        core 42
socket 1        core 43
socket 1        core 44
socket 1        core 45
socket 1        core 46
socket 1        core 47

8.2.2. Picking CPU cores on system enabled for hyperthreading

On nodes where hyperthreading has been enabled, it is important to make sure not to map any other tasks to the core assigned to the sibling threads.

[root@overcloud-compute-0 ~]# for cpu in {12..15}; do cat /sys/devices/system/cpu/"cpu"$cpu/topology/thread_siblings_list; done
12,36
13,37
14,38
15,39

From the above output, if we assign cores 12-15 for PMD threads (NeutronDpdkCoreList), don’t assign CPUs 36-39 to say NovaVcpuPinSet.

8.3. Manual tuning for compute nodes

  • Install tuned2.7.1-3.el7.noarch or later version
  • Edit /etc/tuned/cpu-partitioning-variables.conf to include CPUs to be used for VM vcpus. Do not include CPUs from HostCpusList.
  • Activate with "tuned-adm profile cpu-partitioning"
  • Run "grub2-mkconfig -o /boot/grub2/grub.cfg" to make sure grub is updated
  • Reboot the server

8.3.1. Software and packages required for tuning

Minimal kernel version as well as tuned packages are required for proper tuning to be accomplished.

For the NFV validation lab kernel version used was:

[root@overcloud-compute-0 ~]# uname -a
Linux overcloud-compute-0.localdomain 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Red Hat Enterprise Linux version 7.3 was used:

[root@overcloud-compute-0 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Tuned packages version 2.7.1 were used:

[root@overcloud-compute-0 ~]# rpm -qa | grep tuned
tuned-profiles-cpu-partitioning-2.7.1-5.el7.noarch
tuned-2.7.1-3.el7_3.1.noarch

OVS-DPDK package version 2.6.1 was used:

[root@overcloud-compute-0 instance]# rpm -qa | grep openvs
python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch                                                                                                                                    |
openvswitch-2.6.1-10.git20161206.el7fdb.x86_64                                                                                                                                           |
openstack-neutron-openvswitch-9.2.0-8.el7ost.noarch

8.3.2. Creating cpu-partitioning-variables.conf file and running tuned-adm

After ensuring the correct versions of software and packages are present, the next task is to create the cpu-partitioning-variables.conf file in /etc/tuned directory of the compute nodes:

[root@overcloud-compute-0 ~]# ls /etc/tuned/
active_profile  bootcmdline  cpu-partitioning-variables.conf  tuned-main.conf

[root@overcloud-compute-0 ~]# cat /etc/tuned/cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=12-23

Make the cpu-partitioning configuration active

root@overcloud-compute-0 ~]# tuned-adm profile cpu-partitioning
[root@overcloud-compute-0 ~]# tuned-adm active
Current active profile: cpu-partitioning

Reboot the compute node for the configuration to take effect.

8.3.3. Relocating the emulator threads on compute nodes

To emulate the virtual machine (VNF) on the compute node (hypervisor), CPU(s) are used. It is possible that the CPUs used for PDM threads are picked for this purpose. It is important to relocate them to CPU cores that were not set aside for PMD threads or for VCpuPinSet. The following commands show how this can be done:

First get the instance ID:

[root@overcloud-compute-0 ~]# virsh list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000002              running

Next check which CPUs are currently mapped for this purpose:

[root@overcloud-compute-0 ~]# virsh emulatorpin 1
emulator: CPU Affinity
----------------------------------
       *: 16-23

Relocate emulator pins to use CPU cores 0-5 that we have set aside for housekeep or non-DPDK tasks:

[root@overcloud-compute-0 ~]# virsh emulatorpin 1 0-5

[root@overcloud-compute-0 ~]# virsh emulatorpin 1
emulator: CPU Affinity
----------------------------------
       *: 0-5
Note

This exercise of relocating emulation pins has to be run everytime the guest VM (VNF) has been halted or a new one has been spawned.

8.4. Ensuring compute nodes are properly tuned

Next thing to do is to check to ensure that the manual tuning that was performed on compute nodes has taken effect. This is done by following the steps below.

8.4.1. Visually inspecting files on the compute nodes

cat /etc/tuned/bootcmdline should have the following options:

TUNED_BOOT_CMDLINE="nohz=on nohz_full=12-23 rcu_nocbs=12-23 intel_pstate=disable nosoftlockup"

cat /proc/cmdline:

[root@overcloud-compute-0 ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=UUID=8b30335f-2e9c-4453-8de6-f810f71dfc5e ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet iommu=pt intel_iomm
B hugepagesz=1G hugepages=12 isolcpus=12-23 nohz=on nohz_full=12-23 rcu_nocbs=12-23 intel_pstate=disable nosoftlockup

Check for CPUAffinity:

cat /etc/systemd/system we are seeing ALL CPUs:

CPUAffinity=0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

The above list is what is expected as it omits CPU cores 12-23 (HostIsolatedCoreList) as desired. Next look for irqbalance

cat /etc/sysconfig/irqbalance:

IRQBALANCE_BANNED_CPUS=00fff000

This shows that CPUs 16-23 are banned from being used for housekeeping activities.

8.4.2. Checking for local interrupts (LOC)

On the compute nodes, the CPU cores mapped for PMD threads no more than 1 Local Interrupt should be observed per second. Following bash script grabs local interrupts for 10 seconds on CPU cores 12-15.

[root@overcloud-compute-0 ~]# grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 12"; sleep 10; grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 12"
core 12 250126
core 12 250136;
[root@overcloud-compute-0 ~]# grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 13"; sleep 10; grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 13"
core 13 248473
core 13 248483
[root@overcloud-compute-0 ~]# grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 14"; sleep 10; grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 14"
core 14 248396
core 14 248406
[root@overcloud-compute-0 ~]# grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 15"; sleep 10; grep LOC /proc/interrupts | xargs -L1 | cut -d ':' -f 2 | cut -d 'L' -f 1 | sed s/\\s// | sed s/\\s$// | sed s/\\s/\\n/g | awk '{print "core " NR-1, $0}' | grep "core 15"
core 15 248407
core 15 248417

As it can be seen from above output, cores tuned for service PMD threads only have 10 interrupts in 10 seconds or 1 interrupt per second.

8.5. Tuning on the VNF (guest VM)

8.5.1. CPU partitioning

Since DPDK runs on the guest VM (VNF), we need to ensure that we set aside CPUs for PMD threads for optimal performance.

Firstly, make sure you have the right version of kernel and tuned packages:

[root@vnf1 ~]# uname -a
Linux vnf1.novalocal 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


root@vnf1 ~]# rpm -qa | grep tuned
tuned-2.7.1-3.el7_3.2.noarch
tuned-profiles-cpu-partitioning-2.7.1-5.el7.noarch
Note

Currently Tuned does not included in RHOS10 z2 version, it must be consumed through FDBeta or using manual installation on overcloud images,For RHOS10 z3 Tuned and cpu-partitioning are included.

[root@vnf1 ~]# tuned-adm active
Current active profile: virtual-guest
[root@vnf1 ~]# cd /etc/tuned
[root@vnf1 tuned]#

[root@vnf1 tuned]# ls
active_profile  bootcmdline  cpu-partitioning-variables.conf  tuned-main.conf

We need to add file cpu-partitioning-variables.conf:

[root@overcloud-compute-0 ~]# cat /etc/tuned/cpu-partitioning-variables.conf
# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=1-7

Reboot VM for this to take effect.

After reboot, make sure /proc/cmdline has the right parameters for hugepages and isolcpus:

[root@vnf1 ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-327.22.2.el7.x86_64 root=UUID=2dfbb5e5-1e8c-4d69-bae2-3199b999d800 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8 iommu=pt intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=12 nohz=on nohz_full= rcu_nocbs= intel_pstate=disable nosoftlockup isolcpus=1-7

If it does not have the required parameters use the grubby command to update it: grubby --update-kernel=grubby --default-kernel --args="isolcpus=1-7 default_hugepagesz=1GB hugepagesz=1G hugepages=12"

After running grubby, reboot is required.

8.6. Checking CPU utilization

PMD threads run continuously resulting in CPU cores allocated for these threads to run at 100%. This is true for both the compute nodes as well as the VNFs (VMs). This can be used as an indicator for ensuring tuning was performed correctly. CPU cores that were allocated for PMD threads should show 100% utilization (or close to it) constantly.

8.6.1. CPU utilization on the VNF

On the VNF (VM), run top command and press “1” to show per CPU/core usage. It can be observed that vCPU1 is being used for PMD polling and is being used 100%) as expected. This is shown in Figure 5

fig10 vnf top output

Figure 5: Top Output on the VNF(VM)

8.6.2. CPU utilization on the compute node

Now run top command on the compute node and press “1” to show per CPU/core usage. It can be observed that 12,13,14 and 15 which are tuned to run PMD threads are being used 100% (for user and not for system) as expected shown in Figure 6. Note that CPU17 is also showing 100%. This is because CPU17 corresponds to vCPU1 on the guest VM.

fig9 compute top output

Figure 6: Top output on compute node.

8.7. Troubleshooting and useful commands

8.7.1. OVS-User-Bridge

Use ovs-vsctl show command to display details of the bridges and ports.

root@overcloud-compute-0 ~]# ovs-vsctl show
9ef8aab3-0afa-4fca-9a49-7ca97d3e2ffa
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-link
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure

  Port br-link
            Interface br-link
                type: internal
        Port phy-br-link
            Interface phy-br-link
                type: patch
                options: {peer=int-br-link}
        Port "dpdkbond0"
            Interface "dpdk1"
                type: dpdk
            Interface "dpdk0"
                type: dpdk

8.7.2. Checking if physical NICs are bound to DPDK

By default ports are bound to ixgbe driver. During installation, Red Hat OpenStack director assigns the vfio-pci drivers to the DPDK ports. Use dpdk-devbind -s command to check the status of the DPDK binding

[root@overcloud-compute-0 ~]# dpdk-devbind -s

Network devices using DPDK-compatible driver
============================================
0000:83:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=vfio-pci unused=ixgbe
0000:83:00.1 'Ethernet Controller 10-Gigabit X540-AT2' drv=vfio-pci unused=ixgbe

Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens255f0 drv=ixgbe unused=vfio-pci
0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=ens255f1 drv=ixgbe unused=vfio-pci
0000:04:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens3f0 drv=ixgbe unused=vfio-pci
0000:04:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=ens3f1 drv=ixgbe unused=vfio-pci

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>

From the above output it can be seen that the NICs ens6f0(0000:83:00.0) and ens6f1(0000:83:00.1) are bound to vfio-pci driver (DPDK). If it was bound to ixgbe it would indicate NICs are not bound to DPDK.

8.7.3. Verifying DPDK bond

Since OVS-DPDK bonds are being used, use ovs-appctl bond/show <bond-name> to check status of the bond.

[root@overcloud-compute-0 ~]# ovs-appctl bond/show dpdkbond0
---- dpdkbond0 ----
bond_mode: balance-tcp
bond may use recirculation: yes, Recirc-ID : 1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
next rebalance: 94 ms
lacp_status: negotiated
active slave mac: a0:36:9f:47:e0:62(dpdk1)

slave dpdk0: enabled
        may_enable: true

slave dpdk1: enabled
        active slave
        may_enable: true

It can be seen from the above output that dpdk0 and dpdk1 slaves are enabled on the bond dpdkbond0.

8.7.4. Checking OVS-DPDK counters

Periodically check the counters for receive (rx), transmit (tx), drops and errors (errs) using ovs-ofctl dump-ports <bridge-name> command.

[root@overcloud-compute-0 ~]# ovs-ofctl dump-ports br-link
OFPST_PORT reply (xid=0x2): 4 ports
  port LOCAL: rx pkts=542285, bytes=67242760, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=1, bytes=70, drop=0, errs=0, coll=0
  port  1: rx pkts=153700, bytes=11011978, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=271137, bytes=33620934, drop=0, errs=0, coll=?
  port  2: rx pkts=18084, bytes=2875028, drop=0, errs=0, frame=?, over=?, crc=?
           tx pkts=271135, bytes=33620740, drop=0, errs=0, coll=?
  port  3: rx pkts=0, bytes=0, drop=?, errs=?, frame=?, over=?, crc=?
           tx pkts=0, bytes=0, drop=?, errs=?, coll=?

For each port it is possible to examine transmit and received packets as well as drops. To check for more details and grouping per packets size the following command may be used:

[root@overcloud-compute-0 ~]# ovs-vsctl list Interface|grep -E "^(statistics|name)" | grep -a2 dpdk
name                : "dpdk0"
statistics          : {"rx_128_to_255_packets"=18136, "rx_1_to_64_packets"=1, "rx_256_to_511_packets"=0, "rx_512_to_1023_packets"=0, "rx_65_to_127_packets"=1, rx_broadcast_packets=0, rx_bytes=2883614, rx_dropped=0, rx_errors=0, rx_jabber_errors=0, rx_packets=18138, "tx_128_to_255_packets"=271940, "tx_1_to_64_packets"=0, "tx_256_to_511_packets"=0, "tx_512_to_1023_packets"=0, "tx_65_to_127_packets"=0, tx_broadcast_packets=0, tx_bytes=33720560, tx_dropped=0, tx_errors=0, tx_multicast_packets=271940, tx_packets=271940}
name                : "dpdk1"
< ======= SNIP ============ >