Chapter 8. Configuring OVS TC-flower hardware offload

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Generate an overcloud role for OVS hardware offload that is based on the Compute role:

   Example

   In this example, a role is created, ComputeOvsHwOffload, based on the Compute role. The roles file that the command generates is named, roles_data_compute_ovshwol.yaml:

   $ openstack overcloud roles generate -o \
   roles_data_compute_ovshwol.yaml Controller Compute:ComputeOvsHwOffload

   Note

   If your RHOSP environment includes a mix of OVS-DPDK, SR-IOV, and OVS TC-flower hardware offload technologies, you generate just one roles data file, such as roles_data.yaml to include all the roles:

   $ openstack overcloud roles generate -o /home/stack/templates/\
   roles_data.yaml Controller ComputeOvsDpdk ComputeOvsDpdkSriov \
   Compute:ComputeOvsHwOffload

4. (Optional) change the HostnameFormatDefault: '%stackname%-compute-%index%' name for the ComputeOvsHwOffload role.

5. To generate an images file, you run the openstack tripleo container image prepare command. The following inputs are needed:

   • The roles data file that you generated in an earlier step, for example, roles_data_compute_ovshwol.yaml.

   • The SR-IOV environment file appropriate for your Networking service mechanism driver:

     • ML2/OVN environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml

     • ML2/OVS environments

       /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml

       Example

       In this example, the overcloud_images.yaml file is being generated for an ML2/OVN environment:

       $ sudo openstack tripleo container image prepare \
         --roles-file ~/templates/roles_data_compute_ovshwol.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml \
         -e ~/containers-prepare-parameter.yaml \
         --output-env-file=/home/stack/templates/overcloud_images.yaml

6. Note the path and file name of the roles data file and the images file that you have created. You use these files later when you deploy your overcloud.

Procedure

1. Use one of the following commands on the physical server that contains the PCI cards:

   • If your overcloud is deployed:

     $ lspci -nn -s  <pci_device_address>

     Sample output

     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet
     Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

   • If your overcloud has not been deployed:

     $ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

2. Note the vendor and product IDs for PCI passthrough devices on the ComputeOvsHwOffload nodes. You will need these IDs in a later step.
3. Log in to the undercloud as the stack user.

4. Source the stackrc file:

   $ source ~/stackrc

5. Create a custom environment YAML file, for example, ovshwol-overrides.yaml. Configure the PCI passthrough devices for the compute nodes by adding the following content to the file:

   parameter_defaults:
     NeutronOVSFirewallDriver: iptables_hybrid
     ComputeOvsHwOffloadParameters:
       IsolCpusList: 2-9,21-29,11-19,31-39
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
       OvsHwOffload: true
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-tenant
       NovaPCIPassthrough:
         - vendor_id: <vendor-id>
           product_id: <product-id>
           address: <address>
           physical_network: "tenant"
         - vendor_id: <vendor-id>
           product_id: <product-id>
           address: <address>
           physical_network: "null"
       NovaReservedHostMemory: 4096
       NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39
       ...

   Note

   If you are using Mellanox smart NICs, add DerivePciWhitelistEnabled: true under the ComputeOvsHwOffloadParameters parameter. When using OVS hardware offload, the Compute service (nova) scheduler operates similarly to SR-IOV passthrough for instance spawning.

   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device.
   • Replace <physical_network> with the name of the physical network the PCI device is located on.
   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts.

6. In the custom environment file, ensure that PciPassthroughFilter and NUMATopologyFilter are in the list of filters for the NovaSchedulerEnabledFilters parameter. The Compute service (nova) uses this parameter to filter a node:

   parameter_defaults:
     ...
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter
       - AggregateInstanceExtraSpecsFilter

   Note

   Optional: For details on how to troubleshoot and configure OVS Hardware Offload issues in RHOSP 17.1 with Mellanox ConnectX5 NICs, see Troubleshooting Hardware Offload.

7. Note the path and file name of the custom environment file that you have created. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Open the custom environment YAML file that you created in Section 8.2, “Configuring PCI passthrough devices for OVS TC-flower hardware offload”, or create a new one.

4. Add role-specific parameters for the ComputeOvsHwOffload nodes to the custom environment file.

   Example

     ComputeOvsHwOffloadParameters:
       IsolCpusList: 9-63,73-127
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=9-63,73-127
       NovaReservedHostMemory: 4096
       NovaComputeCpuSharedSet: 0-8,64-72
       NovaComputeCpuDedicatedSet: 9-63,73-127
       TunedProfileName: "cpu-partitioning"

5. Add the OvsHwOffload parameter under role-specific parameters with a value of true.

     ComputeOvsHwOffloadParameters:
       IsolCpusList: 9-63,73-127
       KernelArgs: default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=9-63,73-127
       NovaReservedHostMemory: 4096
       NovaComputeCpuSharedSet: 0-8,64-72
       NovaComputeCpuDedicatedSet: 9-63,73-127
       TunedProfileName: "cpu-partitioning"
       OvsHwOffload: true
     ...

6. Review the configuration defaults that RHOSP director uses to configure OVS hardware offload. These defaults are provided in the file, and they vary based on your mechanism driver:

   • ML2/OVN

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-ovn-sriov.yaml

   • ML2/OVS

     /usr/share/openstack-tripleo-heat-templates/environment/services/neutron-sriov.yaml

7. If you need to override any of the configuration defaults, add your overrides to the custom environment file.

   This custom environment file, for example, is where you can add Nova PCI whitelist values or set the network type.

   Example

   In this example, the Networking service (neutron) network type is set to VLAN and ranges are added for the tenants:

   parameter_defaults:
     NeutronNetworkType: vlan
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

8. If you created a new custom environment file, note its path and file name. You use this file later when you deploy your overcloud.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as overcloud-baremetal-deploy.yaml, as instructed in Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. In the bare metal nodes definition file, add a declaration to the Ansible playbook, cli-overcloud-node-kernelargs.yaml.

   The playbook contains kernel arguments to use when you provision bare metal nodes.

   - name: ComputeOvsHwOffload
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
   ...

5. If you want to set any extra Ansible variables when running the playbook, use the extra_vars property to set them.

   Note

   The variables that you add to extra_vars should be the same role-specific parameters for the ComputeOvsHwOffload nodes that you added to the custom environment file earlier in Section 8.3, “Adding role-specific parameters and configuration overrides for OVS TC-flower hardware offload”.

   Example

   - name: ComputeOvsHwOffload
   ...
     ansible_playbooks:
       - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
         extra_vars:
           kernel_args: 'default_hugepagesz=1GB hugepagesz=1G hugepages=100 amd_iommu=on iommu=pt isolcpus=9-63,73-127'
           tuned_isolated_cores: '9-63,73-127'
           tuned_profile: 'cpu-partitioning'
           reboot_wait_timeout: 1800

6. Note the path and file name of the bare metal nodes definition file that you have created. You use this file later when you configure your NICs and as the input file for the overcloud node provision command when you provision your nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. Copy a sample network configuration template.

   Copy a NIC configuration Jinja2 template from the examples in the /usr/share/ansible/roles/tripleo_network_config/templates/ directory. Choose the one that most closely matches your NIC requirements. Modify it as needed.

4. In your NIC configuration template, for example, single_nic_vlans.j2, add your PF and VF interfaces. To create VFs, configure the interfaces as standalone NICs.

   Example

   ...
   - type: sriov_pf
     name: enp196s0f0np0
     mtu: 9000
     numvfs: 16
     use_dhcp: false
     defroute: false
     nm_controlled: true
     hotplug: true
     promisc: false
     link_mode: switchdev
   ...

   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, the modification might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

5. Add the custom network configuration template to the bare metal nodes definition file that you created in Section 8.4, “Creating a bare metal nodes definition file for OVS TC-flower hardware offload”.

   Example

   - name: ComputeOvsHwOffload
     count: 2
     hostname_format: compute-%index%
     defaults:
       networks:
         - network: internal_api
           subnet: internal_api_subnet
         - network: tenant
           subnet: tenant_subnet
         - network: storage
           subnet: storage_subnet
       network_config:
         template: /home/stack/templates/single_nic_vlans.j2
   ...

6. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Note

   • Do not use the NeutronSriovNumVFs parameter when configuring OVS hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.
7. Note the path and file name of the NIC configuration template that you have created. You use this file later if you want to partition your NICs.

Next steps

1. Provision your overcloud networks.

   For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide

2. Provision your overcloud VIPs.

   For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide

3. Provision your bare metal nodes.

   For more information, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide

4. Deploy your overcloud.

   For more information, see Section 8.6, “Deploying an OVS TC-flower hardware offload overcloud”.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Enter the openstack overcloud deploy command.

   It is important to list the inputs to the openstack overcloud deploy command in a particular order. The general rule is to specify the default heat template files first followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Add your inputs to the openstack overcloud deploy command in the following order:

   1. A custom network definition file that contains the specifications for your SR-IOV network on the overcloud, for example, network-data.yaml.

      For more information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   2. A roles file that contains the Controller and ComputeOvsHwOffload roles that RHOSP director uses to deploy your OVS hardware offload environment.

      Example: roles_data_compute_ovshwol.yaml

      For more information, see Section 8.1, “Generating roles and image files for OVS TC-flower hardware offload”.

   3. An output file from provisioning your overcloud networks.

      Example: overcloud-networks-deployed.yaml

      For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   4. An output file from provisioning your overcloud VIPs.

      Example: overcloud-vip-deployed.yaml

      For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   5. An output file from provisioning bare-metal nodes.

      Example: overcloud-baremetal-deployed.yaml

      For more information, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   6. An images file that director uses to determine where to obtain container images and how to store them.

      Example: overcloud_images.yaml

      For more information, see Section 8.1, “Generating roles and image files for OVS TC-flower hardware offload”.

   7. An environment file for the Networking service (neutron) mechanism driver and router scheme that your environment uses:

      • ML2/OVN

        • Distributed virtual routing (DVR): neutron-ovn-dvr-ha.yaml
        • Centralized virtual routing: neutron-ovn-ha.yaml

      • ML2/OVS

        • Distributed virtual routing (DVR): neutron-ovs-dvr.yaml
        • Centralized virtual routing: neutron-ovs.yaml

   8. An environment file for SR-IOV, depending on your mechanism driver:

      • ML2/OVN

        • neutron-ovn-sriov.yaml

      • ML2/OVS

        • neutron-sriov.yaml

          Note

          If you also have an OVS-DPDK environment, and want to locate OVS-DPDK and SR-IOV instances on the same node, include the following environment files in your deployment script:

          • ML2/OVN

            neutron-ovn-dpdk.yaml

          • ML2/OVS

            neutron-ovs-dpdk.yaml

   9. One or more custom environment files that contain your configuration for:

      • PCI passthrough devices for the ComputeOvsHwOffload nodes.
      • role-specific parameters for the ComputeOvsHwOffload nodes

      • overrides of default configuration values for the OVS hardware offload environment.

        Example: ovshwol-overrides.yaml

        For more information, see:

      • Section 8.2, “Configuring PCI passthrough devices for OVS TC-flower hardware offload”.

      • Section 8.3, “Adding role-specific parameters and configuration overrides for OVS TC-flower hardware offload”.

        Example

        This excerpt from a sample openstack overcloud deploy command demonstrates the proper ordering of the command’s inputs for an SR-IOV, ML2/OVN environment that uses DVR:

        $ openstack overcloud deploy \
        --log-file overcloud_deployment.log \
        --templates /usr/share/openstack-tripleo-heat-templates/ \
        --stack overcloud \
        -n /home/stack/templates/network_data.yaml \
        -r /home/stack/templates/roles_data_compute_ovshwol.yaml \
        -e /home/stack/templates/overcloud-networks-deployed.yaml \
        -e /home/stack/templates/overcloud-vip-deployed.yaml \
        -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
        -e /home/stack/templates/overcloud-images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-dvr-ha.yaml
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/\
        neutron-ovn-sriov.yaml \
        -e /home/stack/templates/ovshwol-overrides.yaml

4. Run the openstack overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.

Next steps

1. Ensure that the e-switch mode for the NICs is set to switchdev.

   The switchdev mode establishes representor ports on the NIC that are mapped to the VFs.

   Important

   You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

   1. Check the NIC by running this command:

      Example

      In this example, the NIC pci/0000:03:00.0 is queried:

      $ sudo devlink dev eswitch show pci/0000:03:00.0

      Sample output

      You should see output similar to the following:

      pci/0000:03:00.0: mode switchdev inline-mode none encap enable

   2. To set the NIC to switchdev mode, run this command:

      Example

      In this example, the e-switch mode for the NIC pci/0000:03:00.0 is set to switchdev:

      $ sudo devlink dev eswitch set pci/0000:03:00.0 mode switchdev

2. To allocate a port from a switchdev-enabled NIC, do the following:

   1. Log in as a RHOSP user with the admin role, and create a neutron port with a binding-profile value of capabilities, and disable port security:

      Important

      You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

      $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 --disable-port-security

   2. Pass this port information when you create the instance.

      You associate the representor port with the instance VF interface and connect the representor port to OVS bridge br-int for one-time OVS data path processing. A VF port representor functions like a software version of a physical “patch panel” front-end.

      For more information about new instance creation, see Section 8.8, “Creating an instance in an SR-IOV or an OVS TC-flower hardware offload environment”.

3. Apply the following configuration on the interfaces, and the representor ports, to ensure that TC Flower pushes the flow programming at the port level:

    $ sudo ethtool -K <device-name> hw-tc-offload on

4. Adjust the number of channels for each network interface to improve performance.

   A channel includes an interrupt request (IRQ) and the set of queues that trigger the IRQ. When you set the mlx5_core driver to switchdev mode, the mlx5_core driver defaults to one combined channel, which might not deliver optimal performance.

   On the physical function (PF) representors, enter the following command to adjust the number of CPUs available to the host.

   Example

   In this example, the number of multi-purpose channels is set to 3 on the network interface, eno3s0f0:

   $ sudo ethtool -L enp3s0f0 combined 3

Procedure

1. Create an aggregate group, and add relevant hosts.

   Define metadata, for example, sriov=true, that matches defined flavor metadata.

   $ openstack aggregate create sriov_group
   $ openstack aggregate add host sriov_group compute-sriov-0.localdomain
   $ openstack aggregate set --property sriov=true sriov_group

2. Create a flavor.

   $ openstack flavor create <flavor> --ram <size_mb> --disk <size_gb> \
   --vcpus <number>

3. Set additional flavor properties.

   Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   $ openstack flavor set --property sriov=true \
   --property hw:cpu_policy=dedicated \
   --property hw:mem_page_size=1GB <flavor>

Procedure

1. Create a flavor.

   $ openstack flavor create <flavor_name> --ram <size_mb> \
   --disk <size_gb> --vcpus <number>

   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in Configuring the Compute service for instance creation.

2. Create the network and the subnet:

   $ openstack network create <network_name> \
   --provider-physical-network tenant \
   --provider-network-type vlan --provider-segment <vlan_id>
   
   $ openstack subnet create <name> --network <network_name> \
   --subnet-range <ip_address_cidr> --dhcp

3. If you are not a RHOSP user with the admin role, your RHOSP administrator can provide you with the necessary VF or PF to create an instance. Proceed to step 5.

4. If you are a RHOSP user with the admin role, you can create VF or PF ports:

   • VF port:

     $ openstack port create --network <network_name> --vnic-type direct \
     --binding-profile '{"capabilities": ["switchdev"]} <port_name>

   • PF port that is dedicated to a single instance:

     This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     $ openstack port create --network <network_name> \
     --vnic-type direct-physical <port_name>

5. Create an instance.

   $ openstack server create --flavor <flavor> --image <image_name> \
   --nic port-id=<id> <instance_name>

Procedure

1. During deployment, use the host network configuration tool os-net-config to enable hw-tc-offload.
2. Enable hw-tc-offload on the sriov_config service any time you reboot the Compute node.

3. Set the hw-tc-offload parameter to on for the NICs that are attached to the bond:.

   Example

   $ ethtool -k ens1f0 | grep tc-offload
   
   hw-tc-offload: on

Procedure

1. Set the eswitch mode to switchdev for the interfaces you use for HW offload.
2. Use the host network configuration tool os-net-config to enable eswitch during deployment.

3. Enable eswitch on the sriov_config service any time you reboot the Compute node.

   Example

   $ devlink dev eswitch show pci/$(ethtool -i ens1f0 | grep bus-info \
   | cut -d ':' -f 2,3,4 | awk '{$1=$1};1')

Procedure

1. To get Broadcom chip table content, use the bcmcmd command.

   $ cl-bcmcmd l2 show

   Sample output

   mac=00:02:00:00:00:08 vlan=2000 GPORT=0x2 modid=0 port=2/xe1
   mac=00:02:00:00:00:09 vlan=2000 GPORT=0x2 modid=0 port=2/xe1 Hit

2. Inspect the Traffic Control (TC) Layer.

   $ tc -s filter show dev p5p1_1 ingress

   Sample output

   …
   filter block 94 protocol ip pref 3 flower chain 5
   filter block 94 protocol ip pref 3 flower chain 5 handle 0x2
     eth_type ipv4
     src_ip 172.0.0.1
     ip_flags nofrag
     in_hw in_hw_count 1
           action order 1: mirred (Egress Redirect to device eth4) stolen
           index 3 ref 1 bind 1 installed 364 sec used 0 sec
           Action statistics:
           Sent 253991716224 bytes 169534118 pkt (dropped 0, overlimits 0 requeues 0)
           Sent software 43711874200 bytes 30161170 pkt
           Sent hardware 210279842024 bytes 139372948 pkt
           backlog 0b 0p requeues 0
           cookie 8beddad9a0430f0457e7e78db6e0af48
           no_percpu

3. Examine the in_hw flags and the statistics in this output. The word hardware indicates that the hardware processes the network traffic. If you use tc-policy=none, you can check this output or a tcpdump to investigate when hardware or software handles the packets. You can see a corresponding log message in dmesg or in ovs-vswitch.log when the driver is unable to offload packets.

4. For Mellanox, as an example, the log entries resemble syndrome messages in dmesg.

   Sample output

   [13232.860484] mlx5_core 0000:3b:00.0: mlx5_cmd_check:756:(pid 131368): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x6b1266)

   In this example, the error code (0x6b1266) represents the following behavior:

   Sample output

   0x6B1266 |  set_flow_table_entry: pop vlan and forward to uplink is not allowed

Procedure

1. Ensure SR-IOV and VT-d are enabled on the system.
2. Enable IOMMU in Linux by adding intel_iommu=on to kernel parameters, for example, using GRUB.

Procedure

1. To enable logging on the offload modules and to get additional log information for this failure, use the following commands on the Compute node:

   ovs-appctl vlog/set dpif_netlink:file:dbg
   # Module name changed recently (check based on the version used
   ovs-appctl vlog/set netdev_tc_offloads:file:dbg [OR] ovs-appctl vlog/set netdev_offload_tc:file:dbg
   ovs-appctl vlog/set tc:file:dbg

2. Inspect the ovs-vswitchd logs again to see additional details about the issue.

   In the following example logs, the offload failed because of an unsupported attribute mark.

    2020-01-31T06:22:11.218Z|00471|dpif_netlink(handler402)|DBG|system@ovs-system: put[create] ufid:61bd016e-eb89-44fc-a17e-958bc8e45fda recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(7),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d2:f5:f3,dst=fa:16:3e:c4:a3:eb),eth_type(0x0800),ipv4(src=10.1.1.8/0.0.0.0,dst=10.1.1.31/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no),icmp(type=0/0,code=0/0), actions:set(tunnel(tun_id=0x3d,src=10.10.141.107,dst=10.10.141.124,ttl=64,tp_dst=4789,flags(df|key))),6
   
   2020-01-31T06:22:11.253Z|00472|netdev_tc_offloads(handler402)|DBG|offloading attribute pkt_mark isn't supported
   
   2020-01-31T06:22:11.257Z|00473|dpif_netlink(handler402)|ERR|failed to offload flow: Operation not supported: p6p1_5