Procedure

1. Log in to the Compute nodes in your RHOSP deployment that have Mellanox NICs that you want to configure.

2. Use the mstflint utility to query the ESWITCH_IPV4_TTL_MODIFY_ENABLE Mellanox firmware parameter .

   [root@compute-1 ~]# yum install -y mstflint
   [root@compute-1 ~]# mstconfig -d <PF PCI BDF> q ESWITCH_IPV4_TTL_MODIFY_ENABLE

3. If the ESWITCH_IPV4_TTL_MODIFY_ENABLE parameter is enabled and set to 1, then set the value to 0 to disable it.

   [root@compute-1 ~]# mstconfig -d <PF PCI BDF> s ESWITCH_IPV4_TTL_MODIFY_ENABLE=0`

4. Reboot the node.

Procedure

1. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted.

   [stack@director ~]$ sudo subscription-manager register

2. Determine the entitlement pool ID for Red Hat OpenStack Platform director, for example {Pool ID} from the following command and output:

   [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             {Pool-ID}-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

3. Include the Pool ID value in the following command to attach the Red Hat OpenStack Platform 17.1 entitlement.

   [stack@director ~]$ sudo subscription-manager attach --pool={Pool-ID}-123456

4. Disable the default repositories.

   subscription-manager repos --disable=*

5. Enable the required repositories for Red Hat OpenStack Platform with NFV.

   $ sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms \
   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.9-for-rhel-9-x86_64-rpms \
   --enable=openstack-for-rhel-9-x86_64-rpms \
   --enable=rhel-9-for-x86_64-nfv-rpms \
   --enable=fast-datapath-for-rhel-9-x86_64-rpms

6. Update your system so you have the latest base system packages.

   [stack@director ~]$ sudo dnf update -y
   [stack@director ~]$ sudo reboot

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_compute_sriov.yaml that includes the Controller and ComputeSriov roles:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_compute_sriov.yaml \
    Controller ComputeSriov

   ComputeSriov is a custom role provided with your RHOSP installation that includes the NeutronSriovAgent, NeutronSriovHostConfig services, in addition to the default compute services.

4. To prepare the SR-IOV containers, include the neutron-sriov.yaml and roles_data_compute_sriov.yaml files when you generate the overcloud_images.yaml file.

   $ sudo openstack tripleo container image prepare \
     --roles-file ~/templates/roles_data_compute_sriov.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml \
     -e ~/containers-prepare-parameter.yaml \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   For more information on container image preparation, see Preparing container images in Installing and managing Red Hat OpenStack Platform with director.

5. Create a copy of the /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml file in your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml /home/stack/templates/network-environment-sriov.yaml

6. Add the following parameters under parameter_defaults in your network-environment-sriov.yaml file to configure the SR-IOV nodes for your cluster and your hardware configuration:

     NeutronNetworkType: 'vlan'
     NeutronNetworkVLANRanges:
       - tenant:22:22
       - tenant:25:25
     NeutronTunnelTypes: ''

7. To determine the vendor_id and product_id for each PCI device type, use one of the following commands on the physical server that has the PCI cards:

   • To return the vendor_id and product_id from a deployed overcloud, use the following command:

     # lspci -nn -s  <pci_device_address>
     3b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [<vendor_id>: <product_id>] (rev 02)

   • To return the vendor_id and product_id of a physical function (PF) if you have not yet deployed the overcloud, use the following command:

     (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection data save <baremetal_node_name> | jq '.inventory.interfaces[] | .name, .vendor, .product'

8. Configure role specific parameters for SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       IsolCpusList: "1-19,21-39"
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=1-19,21-39"
       TunedProfileName: "cpu-partitioning"
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysicalDevMappings:
         - tenant:p7p1
       NovaComputeCpuDedicatedSet: '1-19,21-39'
       NovaReservedHostMemory: 4096

   Note

   The NovaVcpuPinSet parameter is now deprecated, and is replaced by NovaComputeCpuDedicatedSet for dedicated, pinned workloads.

9. Configure the PCI passthrough devices for the SR-IOV compute nodes in your network-environment-sriov.yaml file:

     ComputeSriovParameters:
       ...
       NovaPCIPassthrough:
         - vendor_id: "<vendor_id>"
           product_id: "<product_id>"
           address: <NIC_address>
           physical_network: "<physical_network>"
       ...

   • Replace <vendor_id> with the vendor ID of the PCI device.
   • Replace <product_id> with the product ID of the PCI device.
   • Replace <NIC_address> with the address of the PCI device. For information about how to configure the address parameter, see Guidelines for configuring NovaPCIPassthrough in Configuring the Compute service for instance creation.

   • Replace <physical_network> with the name of the physical network the PCI device is located on.

     Note

     Do not use the devname parameter when you configure PCI passthrough because the device name of a NIC can change. To create a Networking service (neutron) port on a PF, specify the vendor_id, the product_id, and the PCI device address in NovaPCIPassthrough, and create the port with the --vnic-type direct-physical option. To create a Networking service port on a virtual function (VF), specify the vendor_id and product_id in NovaPCIPassthrough, and create the port with the --vnic-type direct option. The values of the vendor_id and product_id parameters might be different between physical function (PF) and VF contexts. For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough in Configuring the Compute service for instance creation.

10. Configure the SR-IOV enabled interfaces in the compute.yaml network configuration template. To create SR-IOV VFs, configure the interfaces as standalone NICs:

                 - type: sriov_pf
                    name: p7p3
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false
    
                  - type: sriov_pf
                    name: p7p4
                    mtu: 9000
                    numvfs: 10
                    use_dhcp: false
                    defroute: false
                    nm_controlled: true
                    hotplug: true
                    promisc: false

    Note

    The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that PF. In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

11. Ensure that the list of default filters includes the value AggregateInstanceExtraSpecsFilter:

    parameter_defaults:
      NovaSchedulerEnabledFilters:
        - AvailabilityZoneFilter
        - ComputeFilter
        - ComputeCapabilitiesFilter
        - ImagePropertiesFilter
        - ServerGroupAntiAffinityFilter
        - ServerGroupAffinityFilter
        - PciPassthroughFilter
        - AggregateInstanceExtraSpecsFilter

12. Run the overcloud_deploy.sh script.

Procedure

1. Open the NIC config file for your chosen role.

2. Add an entry for the interface type sriov_pf to configure a physical function that the host can use:

           - type: sriov_pf
               name: <interface_name>
               use_dhcp: false
               numvfs: <number_of_vfs>
               promisc: <true/false>

   • Replace <interface_name> with the name of the interface.
   • Replace <number_of_vfs> with the number of VFs.
   • Optional: Replace <true/false> with true to set promiscuous mode, or false to disable promiscuous mode. The default value is true.
   Note

   The numvfs parameter replaces the NeutronSriovNumVFs parameter in the network configuration templates. Red Hat does not support modification of the NeutronSriovNumVFs parameter or the numvfs parameter after deployment. If you modify either parameter after deployment, it might cause a disruption for the running instances that have an SR-IOV port on that physical function (PF). In this case, you must hard reboot these instances to make the SR-IOV PCI device available again.

3. Add an entry for the interface type sriov_vf to configure virtual functions that the host can use:

    - type: <bond_type>
      name: internal_bond
      bonding_options: mode=<bonding_option>
      use_dhcp: false
      members:
      - type: sriov_vf
          device: <pf_device_name>
          vfid: <vf_id>
      - type: sriov_vf
          device:  <pf_device_name>
          vfid: <vf_id>
   
    - type: vlan
      vlan_id:
        get_param: InternalApiNetworkVlanID
      spoofcheck: false
      device: internal_bond
      addresses:
      - ip_netmask:
          get_param: InternalApiIpSubnet
      routes:
        list_concat_unique:
        - get_param: InternalApiInterfaceRoutes

   • Replace <bond_type> with the required bond type, for example, linux_bond. You can apply VLAN tags on the bond for other bonds, such as ovs_bond.

   • Replace <bonding_option> with one of the following supported bond modes:

     • active-backup

     • Balance-slb

       Note

       LACP bonds are not supported.

   • Specify the sriov_vf as the interface type to bond in the members section.

     Note

     If you are using an OVS bridge as the interface type, you can configure only one OVS bridge on the sriov_vf of a sriov_pf device. More than one OVS bridge on a single sriov_pf device can result in packet duplication across VFs, and decreased performance.

   • Replace <pf_device_name> with the name of the PF device.
   • If you use a linux_bond, you must assign VLAN tags. If you set a VLAN tag, ensure that you set a unique tag for each VF associated with a single sriov_pf device. You cannot have two VFs from the same PF on the same VLAN.
   • Replace <vf_id> with the ID of the VF. The applicable VF ID range starts at zero, and ends at the maximum number of VFs minus one.
   • Disable spoof checking.
   • Apply VLAN tags on the sriov_vf for linux_bond over VFs.

4. To reserve VFs for instances, include the NovaPCIPassthrough parameter in an environment file, for example:

   NovaPCIPassthrough:
    - address: "0000:19:0e.3"
      trusted: "true"
      physical_network: "sriov1"
    - address: "0000:19:0e.0"
      trusted: "true"
      physical_network: "sriov2"

   Director identifies the host VFs, and derives the PCI addresses of the VFs that are available to the instance.

5. Enable IOMMU on all nodes that require NIC partitioning. For example, if you want NIC Partitioning for Compute nodes, enable IOMMU using the KernelArgs parameter for that role:

   parameter_defaults:
     ComputeParameters:
       KernelArgs: "intel_iommu=on iommu=pt"

   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment.

   For more information, see Configuring manual node reboot to define KernelArgs in Configuring the Compute service for instance creation.

6. Add your role file and environment files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -r os-net-config.yaml
     -e [your environment files] \
     -e /home/stack/templates/<compute_environment_file>.yaml

Validation

1. Log in to the overcloud Compute node as tripleo-admin and check the number of VFs:

   [tripleo-admin@overcloud-compute-0 tripleo-admin]$ sudo cat /sys/class/net/p4p1/device/sriov_numvfs
   10
   [tripleo-admin@overcloud-compute-0 tripleo-admin]$ sudo cat /sys/class/net/p4p2/device/sriov_numvfs
   10

2. Show OVS connections:

   [tripleo-admin@overcloud-compute-0]$ sudo ovs-vsctl show
   b6567fa8-c9ec-4247-9a08-cbf34f04c85f
       Manager "ptcp:6640:127.0.0.1"
           is_connected: true
       Bridge br-sriov2
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port phy-br-sriov2
               Interface phy-br-sriov2
                   type: patch
                   options: {peer=int-br-sriov2}
           Port br-sriov2
               Interface br-sriov2
                   type: internal
       Bridge br-sriov1
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port phy-br-sriov1
               Interface phy-br-sriov1
                   type: patch
                   options: {peer=int-br-sriov1}
           Port br-sriov1
               Interface br-sriov1
                   type: internal
       Bridge br-ex
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port br-ex
               Interface br-ex
                   type: internal
           Port phy-br-ex
               Interface phy-br-ex
                   type: patch
                   options: {peer=int-br-ex}
       Bridge br-tenant
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port br-tenant
               tag: 305
               Interface br-tenant
                   type: internal
           Port phy-br-tenant
               Interface phy-br-tenant
                   type: patch
                   options: {peer=int-br-tenant}
           Port dpdkbond0
               Interface dpdk0
                   type: dpdk
                   options: {dpdk-devargs="0000:18:0e.0"}
               Interface dpdk1
                   type: dpdk
                   options: {dpdk-devargs="0000:18:0a.0"}
       Bridge br-tun
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port vxlan-98140025
               Interface vxlan-98140025
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.37"}
           Port br-tun
               Interface br-tun
                   type: internal
           Port patch-int
               Interface patch-int
                   type: patch
                   options: {peer=patch-tun}
           Port vxlan-98140015
               Interface vxlan-98140015
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.21"}
           Port vxlan-9814009f
               Interface vxlan-9814009f
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.159"}
           Port vxlan-981400cc
               Interface vxlan-981400cc
                   type: vxlan
                   options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="152.20.0.229", out_key=flow, remote_ip="152.20.0.204"}
       Bridge br-int
           Controller "tcp:127.0.0.1:6633"
               is_connected: true
           fail_mode: secure
           datapath_type: netdev
           Port int-br-tenant
               Interface int-br-tenant
                   type: patch
                   options: {peer=phy-br-tenant}
           Port int-br-ex
               Interface int-br-ex
                   type: patch
                   options: {peer=phy-br-ex}
           Port int-br-sriov1
               Interface int-br-sriov1
                   type: patch
                   options: {peer=phy-br-sriov1}
           Port patch-tun
               Interface patch-tun
                   type: patch
                   options: {peer=patch-int}
           Port br-int
               Interface br-int
                   type: internal
           Port int-br-sriov2
               Interface int-br-sriov2
                   type: patch
                   options: {peer=phy-br-sriov2}
           Port vhu4142a221-93
               tag: 1
               Interface vhu4142a221-93
                   type: dpdkvhostuserclient
                   options: {vhost-server-path="/var/lib/vhost_sockets/vhu4142a221-93"}
       ovs_version: "2.13.2"

3. Log in to your OVS-DPDK SR-IOV Compute node as tripleo-admin and check Linux bonds:

   [tripleo-admin@overcloud-computeovsdpdksriov-1 ~]$ cat /proc/net/bonding/<bond_name>
   Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
   
   Bonding Mode: fault-tolerance (active-backup)
   Primary Slave: None
   Currently Active Slave: eno3v1
   MII Status: up
   MII Polling Interval (ms): 0
   Up Delay (ms): 0
   Down Delay (ms): 0
   Peer Notification Delay (ms): 0
   
   Slave Interface: eno3v1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 4e:77:94:bd:38:d2
   Slave queue ID: 0
   
   Slave Interface: eno4v1
   MII Status: up
   Speed: 10000 Mbps
   Duplex: full
   Link Failure Count: 0
   Permanent HW addr: 4a:74:52:a7:aa:7c
   Slave queue ID: 0

4. List OVS bonds:

   [tripleo-admin@overcloud-computeovsdpdksriov-1 ~]$ sudo ovs-appctl bond/show
   ---- dpdkbond0 ----
   bond_mode: balance-slb
   bond may use recirculation: no, Recirc-ID : -1
   bond-hash-basis: 0
   updelay: 0 ms
   downdelay: 0 ms
   next rebalance: 9491 ms
   lacp_status: off
   lacp_fallback_ab: false
   active slave mac: ce:ee:c7:58:8e:b2(dpdk1)
   
   slave dpdk0: enabled
     may_enable: true
   
   slave dpdk1: enabled
     active slave
     may_enable: true

Procedure

1. Generate an overcloud role for OVS hardware offload that is based on the Compute role:

   openstack overcloud roles generate -o roles_data.yaml Controller Compute:ComputeOvsHwOffload

2. Optional: Change the HostnameFormatDefault: '%stackname%-compute-%index%' name for the ComputeOvsHwOffload role.
3. Add the OvsHwOffload parameter under role-specific parameters with a value of true.
4. To configure neutron to use the iptables/hybrid firewall driver implementation, include the line: NeutronOVSFirewallDriver: iptables_hybrid. For more information about NeutronOVSFirewallDriver, see Using the Open vSwitch Firewall in Hardening Red Hat OpenStack Platform.

5. Configure the physical_network parameter to match your environment.

   • For VLAN, set the physical_network parameter to the name of the network you create in neutron after deployment. This value should also be in NeutronBridgeMappings.

   • For VXLAN, set the physical_network parameter to null.

     Example:

     parameter_defaults:
       NeutronOVSFirewallDriver: iptables_hybrid
       ComputeSriovParameters:
         IsolCpusList: 2-9,21-29,11-19,31-39
         KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=128 intel_iommu=on iommu=pt"
         OvsHwOffload: true
         TunedProfileName: "cpu-partitioning"
         NeutronBridgeMappings:
           - tenant:br-tenant
         NovaPCIPassthrough:
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "tenant"
           - vendor_id: <vendor-id>
             product_id: <product-id>
             address: <address>
             physical_network: "null"
         NovaReservedHostMemory: 4096
         NovaComputeCpuDedicatedSet: 1-9,21-29,11-19,31-39

   • Replace <vendor-id> with the vendor ID of the physical NIC.
   • Replace <product-id> with the product ID of the NIC VF.

   • Replace <address> with the address of the physical NIC.

     For more information about how to configure NovaPCIPassthrough, see Guidelines for configuring NovaPCIPassthrough in Configuring the Compute service for instance creation.

6. Ensure that the list of default filters includes NUMATopologyFilter:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   Note

   Optional: For details on how to troubleshoot and configure OVS Hardware Offload issues in RHOSP 17.1 with Mellanox ConnectX5 NICs, see Troubleshooting Hardware Offload.

7. Configure one or more network interfaces intended for hardware offload in the compute-sriov.yaml configuration file:

     - type: ovs_bridge
       name: br-tenant
       mtu: 9000
       members:
       - type: sriov_pf
         name: p7p1
         numvfs: 5
         mtu: 9000
         primary: true
         promisc: true
         use_dhcp: false
         link_mode: switchdev

   Note

   • Do not use the NeutronSriovNumVFs parameter when configuring Open vSwitch hardware offload. The number of virtual functions is specified using the numvfs parameter in a network configuration file used by os-net-config. Red Hat does not support modifying the numvfs setting during update or redeployment.
   • Do not configure Mellanox network interfaces as a nic-config interface type ovs-vlan because this prevents tunnel endpoints such as VXLAN from passing traffic due to driver limitations.

8. Include the ovs-hw-offload.yaml file in the overcloud deploy command:

   TEMPLATES_HOME=”/usr/share/openstack-tripleo-heat-templates”
   CUSTOM_TEMPLATES=”/home/stack/templates”
   
   openstack overcloud deploy --templates \
     -r ${CUSTOM_TEMPLATES}/roles_data.yaml \
     -e ${TEMPLATES_HOME}/environments/ovs-hw-offload.yaml \
     -e ${CUSTOM_TEMPLATES}/network-environment.yaml \
     -e ${CUSTOM_TEMPLATES}/neutron-ovs.yaml

Verification

1. Confirm that a PCI device is in switchdev mode:

   # devlink dev eswitch show pci/0000:03:00.0
   pci/0000:03:00.0: mode switchdev inline-mode none encap enable

2. Verify if offload is enabled in OVS:

   # ovs-vsctl get Open_vSwitch . other_config:hw-offload
   “true”

Procedure

1. To allocate a port from a switchdev-enabled NIC, log in as an admin user, create a neutron port with a binding-profile value of capabilities, and disable port security:

   Important

   You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

   $ openstack port create --network private --vnic-type=direct --binding-profile '{"capabilities": ["switchdev"]}' direct_port1 --disable-port-security

2. Pass this port information when you create the instance.

   You associate the representor port with the instance VF interface and connect the representor port to OVS bridge br-int for one-time OVS data path processing. A VF port representor functions like a software version of a physical “patch panel” front-end.

   For more information about new instance creation, see Deploying an SR-IOV instance.

Procedure

1. Use director to apply the following configuration on OVS:

   $ sudo ovs-vsctl set Open_vSwitch . other_config:hw-offload=true

2. Restart to enable HW Offload.

Procedure

1. Apply the following configuration. This is the default option if you do not explicitly configure tc-policy:

   Important

   You must enable security groups and port security on switchdev ports for the connection tracking (conntrack) module to offload OpenFlow flows to hardware.

   $ sudo ovs-vsctl set Open_vSwitch . other_config:tc-policy=none

2. Restart OVS.

Procedure

1. During deployment, use the host network configuration tool os-net-config to enable hw-tc-offload.
2. Enable hw-tc-offload on the sriov_config service any time you reboot the Compute node.

3. Set the hw-tc-offload parameter to on for the NICs that are attached to the bond:.

   [root@overcloud-computesriov-0 ~]# ethtool -k ens1f0 | grep tc-offload
   hw-tc-offload: on

Procedure

1. Set the eswitch mode to switchdev for the interfaces you use for HW offload.
2. Use the host network configuration tool os-net-config to enable eswitch during deployment.

3. Enable eswitch on the sriov_config service any time you reboot the Compute node.

   [root@overcloud-computesriov-0 ~]# devlink dev eswitch show pci/$(ethtool -i ens1f0 | grep bus-info | cut -d ':' -f 2,3,4 | awk '{$1=$1};1')

Procedure

1. To get Broadcom chip table content, use the bcmcmd command.

   root@dni-7448-26:~# cl-bcmcmd l2 show
   
   mac=00:02:00:00:00:08 vlan=2000 GPORT=0x2 modid=0 port=2/xe1
   mac=00:02:00:00:00:09 vlan=2000 GPORT=0x2 modid=0 port=2/xe1 Hit

2. Inspect the Traffic Control (TC) Layer.

   # tc -s filter show dev p5p1_1 ingress
   …
   filter block 94 protocol ip pref 3 flower chain 5
   filter block 94 protocol ip pref 3 flower chain 5 handle 0x2
     eth_type ipv4
     src_ip 172.0.0.1
     ip_flags nofrag
     in_hw in_hw_count 1
           action order 1: mirred (Egress Redirect to device eth4) stolen
           index 3 ref 1 bind 1 installed 364 sec used 0 sec
           Action statistics:
           Sent 253991716224 bytes 169534118 pkt (dropped 0, overlimits 0 requeues 0)
           Sent software 43711874200 bytes 30161170 pkt
           Sent hardware 210279842024 bytes 139372948 pkt
           backlog 0b 0p requeues 0
           cookie 8beddad9a0430f0457e7e78db6e0af48
           no_percpu

3. Examine the in_hw flags and the statistics in this output. The word hardware indicates that the hardware processes the network traffic. If you use tc-policy=none, you can check this output or a tcpdump to investigate when hardware or software handles the packets. You can see a corresponding log message in dmesg or in ovs-vswitch.log when the driver is unable to offload packets.

4. For Mellanox, as an example, the log entries resemble syndrome messages in dmesg.

   [13232.860484] mlx5_core 0000:3b:00.0: mlx5_cmd_check:756:(pid 131368): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0x6b1266)

   In this example, the error code (0x6b1266) represents the following behavior:

   0x6B1266 |  set_flow_table_entry: pop vlan and forward to uplink is not allowed

Procedure

1. Ensure SR-IOV and VT-d are enabled on the system.
2. Enable IOMMU in Linux by adding intel_iommu=on to kernel parameters, for example, using GRUB.

Procedure

1. To enable logging on the offload modules and to get additional log information for this failure, use the following commands on the Compute node:

   ovs-appctl vlog/set dpif_netlink:file:dbg
   # Module name changed recently (check based on the version used
   ovs-appctl vlog/set netdev_tc_offloads:file:dbg [OR] ovs-appctl vlog/set netdev_offload_tc:file:dbg
   ovs-appctl vlog/set tc:file:dbg

2. Inspect the ovs-vswitchd logs again to see additional details about the issue.

   In the following example logs, the offload failed because of an unsupported attribute mark.

    2020-01-31T06:22:11.218Z|00471|dpif_netlink(handler402)|DBG|system@ovs-system: put[create] ufid:61bd016e-eb89-44fc-a17e-958bc8e45fda recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(7),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=fa:16:3e:d2:f5:f3,dst=fa:16:3e:c4:a3:eb),eth_type(0x0800),ipv4(src=10.1.1.8/0.0.0.0,dst=10.1.1.31/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no),icmp(type=0/0,code=0/0), actions:set(tunnel(tun_id=0x3d,src=10.10.141.107,dst=10.10.141.124,ttl=64,tp_dst=4789,flags(df|key))),6
   
   2020-01-31T06:22:11.253Z|00472|netdev_tc_offloads(handler402)|DBG|offloading attribute pkt_mark isn't supported
   
   2020-01-31T06:22:11.257Z|00473|dpif_netlink(handler402)|ERR|failed to offload flow: Operation not supported: p6p1_5

Procedure

1. Create a flavor.

   $ openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

   Tip

   You can specify the NUMA affinity policy for PCI passthrough devices and SR-IOV interfaces by adding the extra spec hw:pci_numa_affinity_policy to your flavor. For more information, see Flavor metadata in Configuring the Compute service for instance creation.

2. Create the network.

   $ openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   $ openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

3. Create the port.

   • Use vnic-type direct to create an SR-IOV virtual function (VF) port.

     $ openstack port create --network net1 --vnic-type direct sriov_port

   • Use the following command to create a virtual function with hardware offload. You must be an admin user to set --binding-profile.

     $ openstack port create --network net1 --vnic-type direct --binding-profile '{"capabilities": ["switchdev"]} sriov_hwoffload_port

   • Use vnic-type direct-physical to create an SR-IOV physical function (PF) port that is dedicated to a single instance. This PF port is a Networking service (neutron) port but is not controlled by the Networking service, and is not visible as a network adapter because it is a PCI device that is passed through to the instance.

     $ openstack port create --network net1 --vnic-type direct-physical sriov_port

4. Deploy an instance.

   $ openstack server create --flavor <flavor> --image <image> --nic port-id=<id> <instance name>

Procedure

1. You can configure the AggregateInstanceExtraSpecsFilter value, and other necessary filters, through the heat parameter NovaSchedulerEnabledFilters under parameter_defaults in your deployment templates.

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AggregateInstanceExtraSpecsFilter
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   Note

   To add this parameter to the configuration of an existing cluster, you can add it to the heat templates, and run the original deployment script again.

2. Create an aggregate group for SR-IOV, and add relevant hosts. Define metadata, for example, sriov=true, that matches defined flavor metadata.

   # openstack aggregate create sriov_group
   # openstack aggregate add host sriov_group compute-sriov-0.localdomain
   # openstack aggregate set --property sriov=true sriov_group

3. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

4. Set additional flavor properties. Note that the defined metadata, sriov=true, matches the defined metadata on the SR-IOV aggregate.

   # openstack flavor set --property sriov=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB <flavor>

1. If you use composable roles, copy and modify the roles_data.yaml file to add the custom role for OVS-DPDK.
2. Update the appropriate network-environment.yaml file to include parameters for kernel arguments, and DPDK arguments.
3. Update the compute.yaml file to include the bridge for DPDK interface parameters.
4. Update the controller.yaml file to include the same bridge details for DPDK interface parameters.
5. Run the overcloud_deploy.sh script to deploy the overcloud with the DPDK parameters.

1. Set the NeutronGlobalPhysnetMtu parameter in the network-environment.yaml file.

   parameter_defaults:
     # MTU global configuration
     NeutronGlobalPhysnetMtu: 9000

   Note

   Ensure that the OvsDpdkSocketMemory value in the network-environment.yaml file is large enough to support jumbo frames. For more information, see Memory parameters.

2. Set the MTU value on the bridge to the Compute node in the controller.yaml file.

     -
       type: ovs_bridge
       name: br-link0
       use_dhcp: false
       members:
         -
           type: interface
           name: nic3
           mtu: 9000

3. Set the MTU values for an OVS-DPDK bond in the compute.yaml file:

   - type: ovs_user_bridge
     name: br-link0
     use_dhcp: false
     members:
       - type: ovs_dpdk_bond
         name: dpdkbond0
         mtu: 9000
         rx_queue: 2
         members:
           - type: ovs_dpdk_port
             name: dpdk0
             mtu: 9000
             members:
               - type: interface
                 name: nic4
           - type: ovs_dpdk_port
             name: dpdk1
             mtu: 9000
             members:
               - type: interface
                 name: nic5

Procedure

1. In the baremetal_deployment.yaml file or in a custom file, set the following baremetal node pre-provisioning parameters:

   pmd_auto_lb

   Set to true to enable PMD automatic load balancing.

   pmd_load_threshold

   Percentage of processing cycles that one of the PMD threads must use consistently before triggering the PMD load balance. Integer, range 0-100.

   pmd_improvement_threshold

   Minimum percentage of evaluated improvement across the non-isolated PMD threads that triggers a PMD auto load balance. Integer, range 0-100.

   To calculate the estimated improvement, a dry run of the reassignment is done and the estimated load variance is compared with the current variance. The default is 25%.

   pmd_rebal_interval

   Minimum time in minutes between two consecutive PMD Auto Load Balance operations. Range 0-20,000 minutes.

   Configure this value to prevent triggering frequent reassignments where traffic patterns are changeable. For example, you might trigger a reassignment once every 10 minutes or once every few hours.

   Example

   ansible_playbooks:
   …
     - playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-openvswitch-dpdk.yaml
     extra_vars:
      …
      pmd_auto_lb: true
      pmd_load_threshold: "70"
      pmd_improvement_threshold: "25"
      pmd_rebal_interval: "2"

2. Include the baremetal deployment file in your openstack overcloud node provision command as shown in the following example:

   Example

    openstack overcloud node provision \
       --stack overcloud \
       --network-config \
       --templates /usr/share/openstack-tripleo-heat-templates \
       --output ~/templates/overcloud-baremetal-deployed.yaml   \
       home/stack/$OSP17REF/network/baremetal_deployment.yaml

3. In dpdk-config.yaml or a custom file, set the following overcloud deployment parameters:

   OvsPmdAutoLb

   Heat equivalent of pmd_auto_lb.

   Set to true to enable PMD automatic load balancing.

   OvsPmdLoadThreshold

   Heat equivalent of pmd_load_threshold.

   Percentage of processing cycles that one of the PMD threads must use consistently before triggering the PMD load balance. Integer, range 0-100.

   OvsPmdImprovementThreshold

   Heat equivalent of pmd_improvement_threshold.

   Minimum percentage of evaluated improvement across the non-isolated PMD threads that triggers a PMD auto load balance. Integer, range 0-100.

   To calculate the estimated improvement, a dry run of the reassignment is done and the estimated load variance is compared with the current variance. The default is 25%.

   OvsPmdRebalInterval

   Heat equivalent of pmd_rebal_interval.

   Minimum time in minutes between two consecutive PMD Auto Load Balance operations. Range 0-20,000 minutes.

   Configure this value to prevent triggering frequent reassignments where traffic patterns are changeable. For example, you might trigger a reassignment once every 10 minutes or once every few hours.

   Example

   parameter_merge_strategies:
     ComputeOvsDpdkSriovParameters:merge
   …
   parameter_defaults:
     ComputeOvsDpdkSriovParameters:
      …
      OvsPmdAutoLb: true
      OvsPmdLoadThreshold: 70
      OvsPmdImprovementThreshold: 25
      OvsPmdRebalInterval: 2

4. Add dpdk-config.yaml or your dpdk configuration file to the stack with your other environment files, and deploy the overcloud:

   Example

   (undercloud)$ openstack overcloud deploy --templates \ -e [your environment files] \
   -e /home/stack/templates/dpdk-config.yaml

1. Create an aggregate group, and add relevant hosts for OVS-DPDK. Define metadata, for example dpdk=true, that matches defined flavor metadata.

    # openstack aggregate create dpdk_group
    # openstack aggregate add host dpdk_group [compute-host]
    # openstack aggregate set --property dpdk=true dpdk_group

   Note

   Pinned CPU instances can be located on the same Compute node as unpinned instances. For more information, see Configuring CPU pinning on Compute nodes in Configuring the Compute service for instance creation.

2. Create a flavor.

   # openstack flavor create <flavor> --ram <MB> --disk <GB> --vcpus <#>

3. Set flavor properties. Note that the defined metadata, dpdk=true, matches the defined metadata in the DPDK aggregate.

   # openstack flavor set <flavor> --property dpdk=true --property hw:cpu_policy=dedicated --property hw:mem_page_size=1GB --property hw:emulator_threads_policy=isolate

   For details about the emulator threads policy for performance improvements, see Configuring emulator threads in Configuring the Compute service for instance creation.

4. Create the network.

   # openstack network create net1 --provider-physical-network tenant --provider-network-type vlan --provider-segment <VLAN-ID>
   # openstack subnet create subnet1 --network net1 --subnet-range 192.0.2.0/24 --dhcp

5. Optional: If you use multiqueue with OVS-DPDK, set the hw_vif_multiqueue_enabled property on the image that you want to use to create a instance:

   # openstack image set --property hw_vif_multiqueue_enabled=true <image>

6. Deploy an instance.

   # openstack server create --flavor <flavor> --image <glance image> --nic net-id=<network ID> <server_name>

1. Review the bridge configuration, and confirm that the bridge has datapath_type=netdev.

   # ovs-vsctl list bridge br0
   _uuid               : bdce0825-e263-4d15-b256-f01222df96f3
   auto_attach         : []
   controller          : []
   datapath_id         : "00002608cebd154d"
   datapath_type       : netdev
   datapath_version    : "<built-in>"
   external_ids        : {}
   fail_mode           : []
   flood_vlans         : []
   flow_tables         : {}
   ipfix               : []
   mcast_snooping_enable: false
   mirrors             : []
   name                : "br0"
   netflow             : []
   other_config        : {}
   ports               : [52725b91-de7f-41e7-bb49-3b7e50354138]
   protocols           : []
   rstp_enable         : false
   rstp_status         : {}
   sflow               : []
   status              : {}
   stp_enable          : false

2. Optionally, you can view logs for errors, such as if the container fails to start.

   # less /var/log/containers/neutron/openvswitch-agent.log

3. Confirm that the Poll Mode Driver CPU mask of the ovs-dpdk is pinned to the CPUs. In case of hyper threading, use sibling CPUs.

   For example, to check the sibling of CPU4, run the following command:

   # cat /sys/devices/system/cpu/cpu4/topology/thread_siblings_list
   4,20

   The sibling of CPU4 is CPU20, therefore proceed with the following command:

   # ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x100010

   Display the status:

   # tuna -t ovs-vswitchd -CP
   thread  ctxt_switches pid SCHED_ rtpri affinity voluntary nonvoluntary       cmd
   3161	OTHER 	0    	6	765023      	614	ovs-vswitchd
   3219   OTHER 	0    	6     	1        	0   	handler24
   3220   OTHER 	0    	6     	1        	0   	handler21
   3221   OTHER 	0    	6     	1        	0   	handler22
   3222   OTHER 	0    	6     	1        	0   	handler23
   3223   OTHER 	0    	6     	1        	0   	handler25
   3224   OTHER 	0    	6     	1        	0   	handler26
   3225   OTHER 	0    	6     	1        	0   	handler27
   3226   OTHER 	0    	6     	1        	0   	handler28
   3227   OTHER 	0    	6     	2        	0   	handler31
   3228   OTHER 	0    	6     	2        	4   	handler30
   3229   OTHER 	0    	6     	2        	5   	handler32
   3230   OTHER 	0    	6	953538      	431   revalidator29
   3231   OTHER 	0    	6   1424258      	976   revalidator33
   3232   OTHER 	0    	6   1424693      	836   revalidator34
   3233   OTHER 	0    	6	951678      	503   revalidator36
   3234   OTHER 	0    	6   1425128      	498   revalidator35
   *3235   OTHER 	0    	4	151123       	51       	pmd37*
   *3236   OTHER 	0   	20	298967       	48       	pmd38*
   3164   OTHER 	0    	6 	47575        	0  dpdk_watchdog3
   3165   OTHER 	0    	6	237634        	0   vhost_thread1
   3166   OTHER 	0    	6  	3665        	0       	urcu2

Procedure

1. Deploy an overcloud with NovaComputeCpuSharedSet defined for a given role. The value of NovaComputeCpuSharedSet applies to the cpu_shared_set parameter in the nova.conf file for hosts within that role.

   parameter_defaults:
       ComputeOvsDpdkParameters:
           NovaComputeCpuSharedSet: "0-1,16-17"
           NovaComputeCpuDedicatedSet: "2-15,18-31"

2. Create a flavor to build instances with emulator threads separated into a shared pool.

   openstack flavor create --ram <size_mb> --disk <size_gb> --vcpus <vcpus> <flavor>

3. Add the hw:emulator_threads_policy extra specification, and set the value to share. Instances created with this flavor will use the instance CPUs defined in the cpu_share_set parameter in the nova.conf file.

   openstack flavor set <flavor> --property hw:emulator_threads_policy=share

Verification

1. Identify the host and name for a given instance.

   openstack server show <instance_id>

2. Use SSH to log on to the identified host as tripleo-admin.

   ssh tripleo-admin@compute-1
   [compute-1]$ sudo virsh dumpxml instance-00001 | grep `'emulatorpin cpuset'`

1. Add the NeutronPhysicalDevMappings parameter in the parameter_defaults section to link between the logical network name and the physical interface.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2

2. Add the new property, trusted, to the SR-IOV parameters.

   parameter_defaults:
     NeutronPhysicalDevMappings:
       - sriov2:p5p2
     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1572"
         physical_network: "sriov2"
         trusted: "true"

   Note

   You must include double quotation marks around the value "true".

1. Create a network of type vlan.

   openstack network create trusted_vf_network  --provider-network-type vlan \
    --provider-segment 111 --provider-physical-network sriov2 \
    --external --disable-port-security

2. Create a subnet.

   openstack subnet create --network trusted_vf_network \
     --ip-version 4 --subnet-range 192.168.111.0/24 --no-dhcp \
    subnet-trusted_vf_network

3. Create a port. Set the vnic-type option to direct, and the binding-profile option to true.

   openstack port create --network sriov111 \
   --vnic-type direct --binding-profile trusted=true \
   sriov111_port_trusted

4. Create an instance, and bind it to the previously-created trusted port.

   openstack server create --image rhel --flavor dpdk  --network internal --port trusted_vf_network_port_trusted --config-drive True --wait rhel-dpdk-sriov_trusted

1. On the compute node that you created the instance, enter the following command:

   # ip link
   7: p5p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether b4:96:91:1c:40:fa brd ff:ff:ff:ff:ff:ff
       vf 6 MAC fa:16:3e:b8:91:c2, vlan 111, spoof checking off, link-state auto, trust on, query_rss off
       vf 7 MAC fa:16:3e:84:cf:c8, vlan 111, spoof checking off, link-state auto, trust off, query_rss off

2. Verify that the trust status of the VF is trust on. The example output contains details of an environment that contains two ports. Note that vf 6 contains the text trust on.
3. You can disable spoof checking if you set port_security_enabled: false in the Networking service (neutron) network, or if you include the argument --disable-port-security when you run the openstack port create command.

Verification

1. Observe the values for RX queue size and TX queue size in the nova.conf file:

   [libvirt]
   rx_queue_size=1024
   tx_queue_size=1024

2. Check the values for RX queue size and TX queue size in the VM instance XML file generated by libvirt on the Compute host.

   <devices>
      <interface type='vhostuser'>
        <mac address='56:48:4f:4d:5e:6f'/>
        <source type='unix' path='/tmp/vhost-user1' mode='server'/>
        <model type='virtio'/>
        <driver name='vhost' rx_queue_size='1024'   tx_queue_size='1024' />
        <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
      </interface>
   </devices>

3. Verify the values for RX queue size and TX queue size, use the following command on a KVM host:

   $ virsh dumpxml <vm name> | grep queue_size

4. Confirm that the performance has improved.

   You should see values such as 3.8 mpps/core at 0 frame loss.

Procedure

1. Set a new NeutronPhysnetNUMANodesMapping parameter to map the physical network to the NUMA node that you associate with the physical network.

2. If you use tunnels, such as VxLAN or GRE, you must also set the NeutronTunnelNUMANodes parameter.

   parameter_defaults:
     NeutronPhysnetNUMANodesMapping: {<physnet_name>: [<NUMA_NODE>]}
     NeutronTunnelNUMANodes: <NUMA_NODE>,<NUMA_NODE>

   Example

   Here is an example with two physical networks tunneled to NUMA node 0:

   • one project network associated with NUMA node 0

   • one management network without any affinity

     parameter_defaults:
       NeutronBridgeMappings:
         - tenant:br-link0
       NeutronPhysnetNUMANodesMapping: {tenant: [1], mgmt: [0,1]}
       NeutronTunnelNUMANodes: 0

   • In this example, assign the physnet of the device named eno2 to NUMA number 0.

     # ethtool -i eno2
     bus-info: 0000:18:00.1
     
     # cat /sys/devices/pci0000:16/0000:16:02.0/0000:18:00.1/numa_node
     0

     Observe the physnet settings in the example heat template:

     NeutronBridgeMappings: 'physnet1:br-physnet1'
     NeutronPhysnetNUMANodesMapping: {physnet1: [0] }
     
     - type: ovs_user_bridge
                     name: br-physnet1
                     mtu: 9000
                     members:
                       - type: ovs_dpdk_port
                         name: dpdk2
                         members:
                           - type: interface
                             name: eno2

1. Observe the configuration in the file /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf:

   [neutron_physnet_tenant]
   numa_nodes=1
   [neutron_tunnel]
   numa_nodes=1

2. Confirm the new configuration with the lscpu command:

   $ lscpu

3. Launch a VM with the NIC attached to the appropriate network.

Building the real-time image

1. Install the libguestfs-tools package on the undercloud to get the virt-customize tool:

   (undercloud) [stack@undercloud-0 ~]$ sudo dnf install libguestfs-tools

   Important

   If you install the libguestfs-tools package on the undercloud, disable iscsid.socket to avoid port conflicts with the tripleo_iscsid service on the undercloud:

   $ sudo systemctl disable --now iscsid.socket

2. Extract the images:

   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/overcloud-hardened-uefi-full-17.1.x86_64.tar
   (undercloud) [stack@undercloud-0 ~]$ tar -xf /usr/share/rhosp-director-images/ironic-python-agent-17.1.x86_64.tar

3. Copy the default image:

   (undercloud) [stack@undercloud-0 ~]$ cp overcloud-hardened-uefi-full.qcow2 overcloud-realtime-compute.qcow2

4. Register your image to enable Red Hat repositories relevant to your customizations. Replace [username] and [password] with valid credentials in the following example.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager register --username=[username] --password=[password]' \
   subscription-manager release --set 9.0

   Note

   For security, you can remove credentials from the history file if they are used on the command prompt. You can delete individual lines in history using the history -d command followed by the line number.

5. Find a list of pool IDs from your account’s subscriptions, and attach the appropriate pool ID to your image.

   sudo subscription-manager list --all --available | less
   ...
   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'subscription-manager attach --pool [pool-ID]'

6. Add the repositories necessary for Red Hat OpenStack Platform with NFV.

   virt-customize -a overcloud-realtime-compute.qcow2 --run-command \
   'sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-eus-rpms \
   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
   --enable=ansible-2.9-for-rhel-9-x86_64-rpms \
   --enable=rhel-9-for-x86_64-nfv-rpms
   --enable=fast-datapath-for-rhel-9-x86_64-rpms'

7. Create a script to configure real-time capabilities on the image.

   (undercloud) [stack@undercloud-0 ~]$ cat <<'EOF' > rt.sh
     #!/bin/bash
   
     set -eux
   
     dnf -v -y --setopt=protected_packages= erase kernel.$(uname -m)
     dnf -v -y install kernel-rt kernel-rt-kvm tuned-profiles-nfv-host
     grubby --set-default /boot/vmlinuz*rt*
     EOF

8. Run the script to configure the real-time image:

   (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 -v --run rt.sh 2>&1 | tee virt-customize.log

   Note

   If you see the following line in the rt.sh script output, "grubby fatal error: unable to find a suitable template", you can ignore this error.

9. Examine the virt-customize.log file that resulted from the previous command, to check that the packages installed correctly using the rt.sh script .

   (undercloud) [stack@undercloud-0 ~]$ cat virt-customize.log | grep Verifying
   
     Verifying  : kernel-3.10.0-957.el7.x86_64                                 1/1
     Verifying  : 10:qemu-kvm-tools-rhev-2.12.0-18.el7_6.1.x86_64              1/8
     Verifying  : tuned-profiles-realtime-2.10.0-6.el7_6.3.noarch              2/8
     Verifying  : linux-firmware-20180911-69.git85c5d90.el7.noarch             3/8
     Verifying  : tuned-profiles-nfv-host-2.10.0-6.el7_6.3.noarch              4/8
     Verifying  : kernel-rt-kvm-3.10.0-957.10.1.rt56.921.el7.x86_64            5/8
     Verifying  : tuna-0.13-6.el7.noarch                                       6/8
     Verifying  : kernel-rt-3.10.0-957.10.1.rt56.921.el7.x86_64                7/8
     Verifying  : rt-setup-2.0-6.el7.x86_64                                    8/8

10. Relabel SELinux:

    (undercloud) [stack@undercloud-0 ~]$ virt-customize -a overcloud-realtime-compute.qcow2 --selinux-relabel

11. Extract vmlinuz and initrd:

    (undercloud) [stack@undercloud-0 ~]$ mkdir image
    (undercloud) [stack@undercloud-0 ~]$ guestmount -a overcloud-realtime-compute.qcow2 -i --ro image
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/vmlinuz-3.10.0-862.rt56.804.el7.x86_64 ./overcloud-realtime-compute.vmlinuz
    (undercloud) [stack@undercloud-0 ~]$ cp image/boot/initramfs-3.10.0-862.rt56.804.el7.x86_64.img ./overcloud-realtime-compute.initrd
    (undercloud) [stack@undercloud-0 ~]$ guestunmount image

    Note

    The software version in the vmlinuz and initramfs filenames vary with the kernel version.

12. Upload the image:

    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud image upload --update-existing --os-image-name overcloud-realtime-compute.qcow2

1. Create an RT-KVM flavor on the overcloud:

   # openstack flavor create  r1.small 99 4096 20 4
   # openstack flavor set --property hw:cpu_policy=dedicated 99
   # openstack flavor set --property hw:cpu_realtime=yes 99
   # openstack flavor set --property hw:mem_page_size=1GB 99
   # openstack flavor set --property hw:cpu_realtime_mask="^0-1" 99
   # openstack flavor set --property hw:cpu_emulator_threads=isolate 99

2. Launch an RT-KVM instance:

   # openstack server create  --image <rhel> --flavor r1.small --nic net-id=<dpdk-net> test-rt

3. To verify that the instance uses the assigned emulator threads, run the following command:

   # virsh dumpxml <instance-id> | grep vcpu -A1
   <vcpu placement='static'>4</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='1'/>
     <vcpupin vcpu='1' cpuset='3'/>
     <vcpupin vcpu='2' cpuset='5'/>
     <vcpupin vcpu='3' cpuset='7'/>
     <emulatorpin cpuset='0-1'/>
     <vcpusched vcpus='2-3' scheduler='fifo'
     priority='1'/>
   </cputune>

1. Add the custom resources for OVS-DPDK under resource_registry:

     resource_registry:
       # Specify the relative/absolute path to the config files you want to use for override the default.
       OS::TripleO::ComputeOvsDpdkSriov::Net::SoftwareConfig: nic-configs/computeovsdpdksriov.yaml
       OS::TripleO::Controller::Net::SoftwareConfig: nic-configs/controller.yaml

2. Under parameter_defaults, set the tunnel type to vxlan, and the network type to vxlan,vlan:

   NeutronTunnelTypes: 'vxlan'
   NeutronNetworkType: 'vxlan,vlan'

3. Under parameters_defaults, set the bridge mapping:

   # The OVS logical->physical bridge mappings to use.
   NeutronBridgeMappings:
     - dpdk-mgmt:br-link0

4. Under parameter_defaults, set the role-specific parameters for the ComputeOvsDpdkSriov role:

     ##########################
     # OVS DPDK configuration #
     ##########################
     ComputeOvsDpdkSriovParameters:
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2-19,22-39"
       TunedProfileName: "cpu-partitioning"
       IsolCpusList: "2-19,22-39"
       NovaComputeCpuDedicatedSet: ['4-19,24-39']
       NovaReservedHostMemory: 4096
       OvsDpdkSocketMemory: "3072,1024"
       OvsDpdkMemoryChannels: "4"
       OvsPmdCoreList: "2,22,3,23"
       NovaComputeCpuSharedSet: [0,20,1,21]
       NovaLibvirtRxQueueSize: 1024
       NovaLibvirtTxQueueSize: 1024

   Note

   To prevent failures during guest creation, assign at least one CPU with sibling thread on each NUMA node. In the example, the values for the OvsPmdCoreList parameter denote cores 2 and 22 from NUMA 0, and cores 3 and 23 from NUMA 1.

   Note

   These huge pages are consumed by the virtual machines, and also by OVS-DPDK using the OvsDpdkSocketMemory parameter as shown in this procedure. The number of huge pages available for the virtual machines is the boot parameter minus the OvsDpdkSocketMemory.

   You must also add hw:mem_page_size=1GB to the flavor you associate with the DPDK instance.

   Note

   OvsDpdkMemoryChannels is a required setting for this procedure. For optimum operation, ensure you deploy DPDK with appropriate parameters and values.

5. Configure the role-specific parameters for SR-IOV:

     NovaPCIPassthrough:
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.0"
         trusted: "true"
         physical_network: "sriov-1"
       - vendor_id: "8086"
         product_id: "1528"
         address: "0000:06:00.1"
         trusted: "true"
         physical_network: "sriov-2"

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic2
         primary: true

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageMgmtNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageMgmtIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: ExternalNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: ExternalIpSubnet
       routes:
       - default: true
         next_hop:
           get_param: ExternalInterfaceDefaultRoute

3. Create the OVS bridge to access neutron-dhcp-agent and neutron-metadata-agent services.

     - type: ovs_bridge
       name: br-link0
       use_dhcp: false
       mtu: 9000
       members:
       - type: interface
         name: nic3
         mtu: 9000
       - type: vlan
         vlan_id:
           get_param: TenantNetworkVlanID
         mtu: 9000
         addresses:
         - ip_netmask:
             get_param: TenantIpSubnet

1. Create the control-plane Linux bond for an isolated network.

     - type: linux_bond
       name: bond_api
       bonding_options: "mode=active-backup"
       use_dhcp: false
       dns_servers:
         get_param: DnsServers
       members:
       - type: interface
         name: nic3
         primary: true
       - type: interface
         name: nic4

2. Assign VLANs to this Linux bond.

     - type: vlan
       vlan_id:
         get_param: InternalApiNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: InternalApiIpSubnet
   
     - type: vlan
       vlan_id:
         get_param: StorageNetworkVlanID
       device: bond_api
       addresses:
       - ip_netmask:
           get_param: StorageIpSubnet

3. Set a bridge with a DPDK port to link to the controller.

     - type: ovs_user_bridge
       name: br-link0
       use_dhcp: false
       ovs_extra:
         - str_replace:
             template: set port br-link0 tag=_VLAN_TAG_
             params:
               _VLAN_TAG_:
                  get_param: TenantNetworkVlanID
       addresses:
         - ip_netmask:
             get_param: TenantIpSubnet
       members:
         - type: ovs_dpdk_bond
           name: dpdkbond0
           mtu: 9000
           rx_queue: 2
           members:
             - type: ovs_dpdk_port
               name: dpdk0
               members:
                 - type: interface
                   name: nic7
             - type: ovs_dpdk_port
               name: dpdk1
               members:
                 - type: interface
                   name: nic8

   Note

   To include multiple DPDK devices, repeat the type code section for each DPDK device that you want to add.

   Note

   When using OVS-DPDK, all bridges on the same Compute node must be of type ovs_user_bridge. Red Hat OpenStack Platform does not support both ovs_bridge and ovs_user_bridge located on the same node.