Warning message

Log in to add comments or rate this document.

SR-IOV with OVS Hardware Offload validation and supportibilty for TC offloading

Updated 2019-05-23T06:11:48+00:00

Defining the Architecture


  • In this example, the SRIOV with OVS hardware offloading compute node validated with Mellanox Connentx4 LX PCI card.

SRIOV with OVS Hardware Offload validation

  • Post-deployment, from the compute node, verify OpenvSwitch enabled with hardware offload flag and ensure PF enabled with switchdev mode and bind with VFs.


    # ovs-vsctl -t 5 get Open_vSwitch . other_config
    $ /usr/sbin/devlink dev eswitch show pci/0000:03:00.0
    pci/0000:03:00.0: mode switchdev inline-mode transport encap enable
    $ cat /sys/class/net/ens2f0/device/sriov_numvfs
    $ grep ^[^#] var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep passthrough_whitelist
  • Ensure the neutron ports should create with a switchdev binding profile like below example.

    # openstack port create --network <network-name> --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' <neutron-port-name>
    $ openstack port show <neutron-port-name>
    | Field                 | Value                                                                                                              |
    | admin_state_up        | UP                                                                                                                 |
    | allowed_address_pairs |                                                                                                                    |
    | binding_host_id       | overcloud-compute-0.localdomain                                                                                    |
    | binding_profile       | capabilities='[u'switchdev']', pci_slot='0000:03:00.2', pci_vendor_info='15b3:1016', physical_network='datacentre' |
    | binding_vif_details   | datapath_type='system', ovs_hybrid_plug='False', port_filter='True'                                                |
    | binding_vif_type      | ovs                                                                                                                |
    | binding_vnic_type     | direct                                                                                                             |
    | mac_address           | aa:aa:aa:aa:aa:aa                                                                                                  |
    # ip link show ens2f0
    6: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
        link/ether bb:bb:bb:bb:bb:bb brd ff:ff:ff:ff:ff:ff promiscuity 1 
        vf 0 MAC aa:aa:aa:aa:aa:aa, vlan 797, spoof checking off, link-state enable, trust off, query_rss off
  • The neutron port should bind with OpenvSwitch integretion bridge (br-int) with sudo name (e.g., eth0, eth1, eth2) and PF (e.g., ens2f0) bind with provider bridge (e.g., br-data).

    # ovs-vsctl show
        Manager "ptcp:6640:"
            is_connected: true
        Bridge br-int
            Controller "tcp:"
                is_connected: true
            fail_mode: secure
            Port "eth1"
                tag: 2
                Interface "eth1"
            Port "eth0"
                tag: 2
                Interface "eth0"
        Bridge br-data
            Controller "tcp:"
                is_connected: true
            Port "ens2f0"
                Interface "ens2f0"
            Port br-data
                Interface br-data
                    type: internal
        ovs_version: "2.9.0"
  • Review OpenvSwitch port details in data path layer.


    # ovs-dpctl show
        lookups: hit:7546 missed:87300 lost:0
        flows: 6
        masks: hit:114758 total:4 hit/pkt:1.21
        port 0: ovs-system (internal)
        port 1: br-data (internal)
        port 2: ens2f0
        port 3: br-int (internal)
        port 4: eth0
        port 5: eth1
        port 6: eth2

Supportability for TC offloading

  • Currently, TC offloading may is not fully supported and everything is related to connection tracking is not offloadable.
    Below reference journal logs.

    Mar 28 06:02:15 overcloud-computesriov-0 ovs-vsctl[36637]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
    Mar 28 06:02:15 overcloud-computesriov-0 kernel: TECH PREVIEW: TC offloading may not be fully supported.
                                                     Please review the provided documentation for limitations.
  • If ML2-OVS configured with openvswitch firewall driver and neutron port of VM enable with the security group, then ovs-vswitchd log flooded with below error message and during traffic communication, the flow related to VF not getting offloaded to PF.

    2019-03-29T07:18:32.156Z|142248|ofproto_dpif_xlate(handler27)|DBG|bridge br-int: learned that ff:ff:ff:ff:ff:ff is on port eth0 in VLAN 1
    2019-03-29T07:18:32.156Z|142249|ofproto_dpif_xlate(handler27)|DBG|bridge br-data: learned that ff:ff:ff:ff:ff:ff is on port phy-br-data in VLAN 111
    2019-03-29T07:18:32.156Z|142250|netdev_tc_offloads(handler27)|DBG|offloading attribute ct_state isn't supported
    2019-03-29T07:18:32.156Z|142251|dpif_netlink(handler27)|ERR|failed to offload flow: Operation not supported

    Note: There is downstream bugzilla 1694545 track for move the ERR trace to WARN.

    # ovs-dpctl dump-flows type=offloaded
    in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ce:ce:ce:ce:ce:ce),eth_type(0x88cc), packets:0, bytes:0, used:4.840s, actions:drop
    in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ab:ab:ab:ab:ab:ab),eth_type(0/0xffff), packets:0, bytes:0, used:1.070s, actions:drop
  • For the workaround to offload all flows, currently it is recommended to disable the security group --disable-port-security while creating the OpenStack network.

    usage: openstack network create [-h] [-f {json,shell,table,value,yaml}]
                                    [-c COLUMN] [--max-width <integer>]
                                    [--fit-width] [--print-empty] [--noindent]
                                    [--prefix PREFIX] [--share | --no-share]
                                    [--enable | --disable] [--project <project>]
                                    [--description <description>] [--mtu <mtu>]
                                    [--project-domain <project-domain>]
                                    [--availability-zone-hint <availability-zone>]
                                    [--enable-port-security | --disable-port-security]
                                    [--external | --internal]
                                    [--default | --no-default]
                                    [--qos-policy <qos-policy>]
                                    [--transparent-vlan | --no-transparent-vlan]
                                    [--provider-network-type <provider-network-type>]
                                    [--provider-physical-network <provider-physical-network>]
                                    [--provider-segment <provider-segment>]
                                    [--tag <tag> | --no-tag]
  • Once the security group disabled for neutron port, then the all associated flow get offloaded during traffic communication.

    # ovs-dpctl dump-flows type=offloaded
    in_port(2),eth(src=dd:dd:dd:dd:dd:dd,dst=ab:ab:ab:ab:ab:ab),eth_type(0/0xffff), packets:0, bytes:0, used:8.260s, actions:drop
    in_port(5),eth(src=aa:aa:aa:aa:aa:aa,dst=cc:cc:cc:cc:cc:cc),eth_type(0x0800),ipv4(frag=no), packets:12, bytes:1176, used:0.990s, actions:4
    in_port(5),eth(src=aa:aa:aa:aa:aa:aa,dst=cc:cc:cc:cc:cc:cc),eth_type(0x0806), packets:1, bytes:60, used:7.260s, actions:4
    in_port(4),eth(src=cc:cc:cc:cc:cc:cc,dst=aa:aa:aa:aa:aa:aa),eth_type(0x0800),ipv4(frag=no), packets:12, bytes:1176, used:0.990s, actions:5
    in_port(4),eth(src=cc:cc:cc:cc:cc:cc,dst=aa:aa:aa:aa:aa:aa),eth_type(0x0806), packets:0, bytes:0, used:8.780s, actions:5