SR-IOV with OVS Hardware Offload validation and supportibilty for TC offloading

Defining the Architecture

• In this example, the SRIOV with OVS hardware offloading compute node validated with Mellanox Connentx4 LX PCI card.

SRIOV with OVS Hardware Offload validation

• Post-deployment, from the compute node, verify OpenvSwitch enabled with hardware offload flag and ensure PF enabled with switchdev mode and bind with VFs.

  Example:

  # ovs-vsctl -t 5 get Open_vSwitch . other_config
  {hw-offload="true"}
  

  $ /usr/sbin/devlink dev eswitch show pci/0000:03:00.0
  pci/0000:03:00.0: mode switchdev inline-mode transport encap enable
  

  $ cat /sys/class/net/ens2f0/device/sriov_numvfs
  4
  

  $ grep ^[^#] var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep passthrough_whitelist
  passthrough_whitelist={"devname":"ens2f0","physical_network":"datacentre"}
  

• Ensure the neutron ports should create with a switchdev binding profile like below example.

  # openstack port create --network <network-name> --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' <neutron-port-name>
  

  $ openstack port show <neutron-port-name>
  +-----------------------+--------------------------------------------------------------------------------------------------------------------+
  | Field                 | Value                                                                                                              |
  +-----------------------+--------------------------------------------------------------------------------------------------------------------+
  | admin_state_up        | UP                                                                                                                 |
  | allowed_address_pairs |                                                                                                                    |
  | binding_host_id       | overcloud-compute-0.localdomain                                                                                    |
  | binding_profile       | capabilities='[u'switchdev']', pci_slot='0000:03:00.2', pci_vendor_info='15b3:1016', physical_network='datacentre' |
  | binding_vif_details   | datapath_type='system', ovs_hybrid_plug='False', port_filter='True'                                                |
  | binding_vif_type      | ovs                                                                                                                |
  | binding_vnic_type     | direct                                                                                                             |
  <..trim..>
  | mac_address           | aa:aa:aa:aa:aa:aa                                                                                                  |
  <..trim..>
  +-----------------------+--------------------------------------------------------------------------------------------------------------------+
  

  # ip link show ens2f0
  6: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
      link/ether bb:bb:bb:bb:bb:bb brd ff:ff:ff:ff:ff:ff promiscuity 1 
      vf 0 MAC aa:aa:aa:aa:aa:aa, vlan 797, spoof checking off, link-state enable, trust off, query_rss off
      <..trim..>
  

• The neutron port should bind with OpenvSwitch integretion bridge (br-int) with sudo name (e.g., eth0, eth1, eth2) and PF (e.g., ens2f0) bind with provider bridge (e.g., br-data).
  Example:

  # ovs-vsctl show
      Manager "ptcp:6640:127.0.0.1"
          is_connected: true
      Bridge br-int
          Controller "tcp:127.0.0.1:6633"
              is_connected: true
          fail_mode: secure
          Port "eth1"
              tag: 2
              Interface "eth1"
  <..trim>
          Port "eth0"
              tag: 2
              Interface "eth0"
      Bridge br-data
          Controller "tcp:127.0.0.1:6633"
              is_connected: true
  <..trim..>
          Port "ens2f0"
              Interface "ens2f0"
          Port br-data
              Interface br-data
                  type: internal
      ovs_version: "2.9.0"
  

• Review OpenvSwitch port details in data path layer.

  Example:

  # ovs-dpctl show
  system@ovs-system:
      lookups: hit:7546 missed:87300 lost:0
      flows: 6
      masks: hit:114758 total:4 hit/pkt:1.21
      port 0: ovs-system (internal)
      port 1: br-data (internal)
      port 2: ens2f0
      port 3: br-int (internal)
      port 4: eth0
      port 5: eth1
      port 6: eth2
  

Supportability for TC offloading

• Currently, TC offloading may is not fully supported and everything is related to connection tracking is not offloadable.
  Below reference journal logs.

  Mar 28 06:02:15 overcloud-computesriov-0 ovs-vsctl[36637]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
  Mar 28 06:02:15 overcloud-computesriov-0 kernel: TECH PREVIEW: TC offloading may not be fully supported.
                                                   Please review the provided documentation for limitations.
  

• If ML2-OVS configured with openvswitch firewall driver and neutron port of VM enable with the security group, then ovs-vswitchd log flooded with below error message and during traffic communication, the flow related to VF not getting offloaded to PF.

  2019-03-29T07:18:32.156Z|142248|ofproto_dpif_xlate(handler27)|DBG|bridge br-int: learned that ff:ff:ff:ff:ff:ff is on port eth0 in VLAN 1
  2019-03-29T07:18:32.156Z|142249|ofproto_dpif_xlate(handler27)|DBG|bridge br-data: learned that ff:ff:ff:ff:ff:ff is on port phy-br-data in VLAN 111
  2019-03-29T07:18:32.156Z|142250|netdev_tc_offloads(handler27)|DBG|offloading attribute ct_state isn't supported
  2019-03-29T07:18:32.156Z|142251|dpif_netlink(handler27)|ERR|failed to offload flow: Operation not supported
  

  Note: There is downstream bugzilla 1694545 track for move the ERR trace to WARN.

  # ovs-dpctl dump-flows type=offloaded
  in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ce:ce:ce:ce:ce:ce),eth_type(0x88cc), packets:0, bytes:0, used:4.840s, actions:drop
  in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ab:ab:ab:ab:ab:ab),eth_type(0/0xffff), packets:0, bytes:0, used:1.070s, actions:drop
  

• For the workaround to offload all flows, currently it is recommended to disable the security group --disable-port-security while creating the OpenStack network.

  usage: openstack network create [-h] [-f {json,shell,table,value,yaml}]
                                  [-c COLUMN] [--max-width <integer>]
                                  [--fit-width] [--print-empty] [--noindent]
                                  [--prefix PREFIX] [--share | --no-share]
                                  [--enable | --disable] [--project <project>]
                                  [--description <description>] [--mtu <mtu>]
                                  [--project-domain <project-domain>]
                                  [--availability-zone-hint <availability-zone>]
                                  [--enable-port-security | --disable-port-security]
                                  [--external | --internal]
                                  [--default | --no-default]
                                  [--qos-policy <qos-policy>]
                                  [--transparent-vlan | --no-transparent-vlan]
                                  [--provider-network-type <provider-network-type>]
                                  [--provider-physical-network <provider-physical-network>]
                                  [--provider-segment <provider-segment>]
                                  [--tag <tag> | --no-tag]
                                  <name>
  

• Once the security group disabled for neutron port, then the all associated flow get offloaded during traffic communication.

  # ovs-dpctl dump-flows type=offloaded
  in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ab:ab:ab:ab:ab:ab),eth_type(0/0xffff), packets:0, bytes:0, used:8.260s, actions:drop
  in_port(5),eth(src=aa:aa:aa:aa:aa:aa,dst=cc:cc:cc:cc:cc:cc),eth_type(0x0800),ipv4(frag=no), packets:12, bytes:1176, used:0.990s, actions:4
  in_port(5),eth(src=aa:aa:aa:aa:aa:aa,dst=cc:cc:cc:cc:cc:cc),eth_type(0x0806), packets:1, bytes:60, used:7.260s, actions:4
  in_port(4),eth(src=cc:cc:cc:cc:cc:cc,dst=aa:aa:aa:aa:aa:aa),eth_type(0x0800),ipv4(frag=no), packets:12, bytes:1176, used:0.990s, actions:5
  in_port(4),eth(src=cc:cc:cc:cc:cc:cc,dst=aa:aa:aa:aa:aa:aa),eth_type(0x0806), packets:0, bytes:0, used:8.780s, actions:5