Chapter 14. Sample DPDK SR-IOV YAML and Jinja2 files

This section provides sample yaml files as a reference to add single root I/O virtualization (SR-IOV) and Data Plane Development Kit (DPDK) interfaces on the same compute node.

Note

These templates are from a fully-configured environment, and include parameters unrelated to NFV, that might not apply to your deployment. For a list of component support levels, see the Red Hat Knowledgebase solution Component Support Graduation.

14.1. roles_data.yaml

  • Run the openstack overcloud roles generate command to generate the roles_data.yaml file.

    Include role names in the command according to the roles that you want to deploy in your environment, such as Controller, ComputeSriov, ComputeOvsDpdkRT, ComputeOvsDpdkSriov, or other roles.

    Example

    For example, to generate a roles_data.yaml file that contains the roles Controller and ComputeHCIOvsDpdkSriov, run the following command:

    $ openstack overcloud roles generate -o roles_data.yaml \
    Controller ComputeHCIOvsDpdkSriov
    ###############################################################################
    # File generated by TripleO
    ###############################################################################
    ###############################################################################
    # Role: Controller                                                            #
    ###############################################################################
    - name: Controller
      description: |
        Controller role that has all the controller services loaded and handles
        Database, Messaging and Network functions.
      CountDefault: 1
      tags:
        - primary
        - controller
      networks:
        External:
          subnet: external_subnet
        InternalApi:
          subnet: internal_api_subnet
        Storage:
          subnet: storage_subnet
        StorageMgmt:
          subnet: storage_mgmt_subnet
        Tenant:
          subnet: tenant_subnet
      # For systems with both IPv4 and IPv6, you may specify a gateway network for
      # each, such as ['ControlPlane', 'External']
      default_route_networks: ['External']
      HostnameFormatDefault: '%stackname%-controller-%index%'
      # Deprecated & backward-compatible values (FIXME: Make parameters consistent)
      # Set uses_deprecated_params to True if any deprecated params are used.
      uses_deprecated_params: True
      deprecated_param_extraconfig: 'controllerExtraConfig'
      deprecated_param_flavor: 'OvercloudControlFlavor'
      deprecated_param_image: 'controllerImage'
      deprecated_nic_config_name: 'controller.yaml'
      update_serial: 1
      ServicesDefault:
        - OS::TripleO::Services::Aide
        - OS::TripleO::Services::AodhApi
        - OS::TripleO::Services::AodhEvaluator
        - OS::TripleO::Services::AodhListener
        - OS::TripleO::Services::AodhNotifier
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::BarbicanApi
        - OS::TripleO::Services::BarbicanBackendSimpleCrypto
        - OS::TripleO::Services::BarbicanBackendDogtag
        - OS::TripleO::Services::BarbicanBackendKmip
        - OS::TripleO::Services::BarbicanBackendPkcs11Crypto
        - OS::TripleO::Services::BootParams
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CeilometerAgentCentral
        - OS::TripleO::Services::CeilometerAgentNotification
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CephGrafana
        - OS::TripleO::Services::CephMds
        - OS::TripleO::Services::CephMgr
        - OS::TripleO::Services::CephMon
        - OS::TripleO::Services::CephRbdMirror
        - OS::TripleO::Services::CephRgw
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::CinderApi
        - OS::TripleO::Services::CinderBackendDellPs
        - OS::TripleO::Services::CinderBackendDellSc
        - OS::TripleO::Services::CinderBackendDellEMCPowermax
        - OS::TripleO::Services::CinderBackendDellEMCPowerStore
        - OS::TripleO::Services::CinderBackendDellEMCSc
        - OS::TripleO::Services::CinderBackendDellEMCUnity
        - OS::TripleO::Services::CinderBackendDellEMCVMAXISCSI
        - OS::TripleO::Services::CinderBackendDellEMCVNX
        - OS::TripleO::Services::CinderBackendDellEMCVxFlexOS
        - OS::TripleO::Services::CinderBackendDellEMCXtremio
        - OS::TripleO::Services::CinderBackendDellEMCXTREMIOISCSI
        - OS::TripleO::Services::CinderBackendNetApp
        - OS::TripleO::Services::CinderBackendPure
        - OS::TripleO::Services::CinderBackendScaleIO
        - OS::TripleO::Services::CinderBackendVRTSHyperScale
        - OS::TripleO::Services::CinderBackendNVMeOF
        - OS::TripleO::Services::CinderBackup
        - OS::TripleO::Services::CinderHPELeftHandISCSI
        - OS::TripleO::Services::CinderScheduler
        - OS::TripleO::Services::CinderVolume
        - OS::TripleO::Services::Clustercheck
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::ContainerImagePrepare
        - OS::TripleO::Services::DesignateApi
        - OS::TripleO::Services::DesignateCentral
        - OS::TripleO::Services::DesignateProducer
        - OS::TripleO::Services::DesignateWorker
        - OS::TripleO::Services::DesignateMDNS
        - OS::TripleO::Services::DesignateSink
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::Ec2Api
        - OS::TripleO::Services::Etcd
        - OS::TripleO::Services::ExternalSwiftProxy
        - OS::TripleO::Services::GlanceApi
        - OS::TripleO::Services::GnocchiApi
        - OS::TripleO::Services::GnocchiMetricd
        - OS::TripleO::Services::GnocchiStatsd
        - OS::TripleO::Services::HAproxy
        - OS::TripleO::Services::HeatApi
        - OS::TripleO::Services::HeatApiCloudwatch
        - OS::TripleO::Services::HeatApiCfn
        - OS::TripleO::Services::HeatEngine
        - OS::TripleO::Services::Horizon
        - OS::TripleO::Services::IpaClient
        - OS::TripleO::Services::Ipsec
        - OS::TripleO::Services::IronicApi
        - OS::TripleO::Services::IronicConductor
        - OS::TripleO::Services::IronicInspector
        - OS::TripleO::Services::IronicPxe
        - OS::TripleO::Services::IronicNeutronAgent
        - OS::TripleO::Services::Iscsid
        - OS::TripleO::Services::Keepalived
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::Keystone
        - OS::TripleO::Services::LoginDefs
        - OS::TripleO::Services::ManilaApi
        - OS::TripleO::Services::ManilaBackendCephFs
        - OS::TripleO::Services::ManilaBackendIsilon
        - OS::TripleO::Services::ManilaBackendNetapp
        - OS::TripleO::Services::ManilaBackendUnity
        - OS::TripleO::Services::ManilaBackendVNX
        - OS::TripleO::Services::ManilaBackendVMAX
        - OS::TripleO::Services::ManilaScheduler
        - OS::TripleO::Services::ManilaShare
        - OS::TripleO::Services::Memcached
        - OS::TripleO::Services::MetricsQdr
        - OS::TripleO::Services::MistralApi
        - OS::TripleO::Services::MistralEngine
        - OS::TripleO::Services::MistralExecutor
        - OS::TripleO::Services::MistralEventEngine
        - OS::TripleO::Services::Multipathd
        - OS::TripleO::Services::MySQL
        - OS::TripleO::Services::MySQLClient
        - OS::TripleO::Services::NeutronApi
        - OS::TripleO::Services::NeutronBgpVpnApi
        - OS::TripleO::Services::NeutronSfcApi
        - OS::TripleO::Services::NeutronCorePlugin
        - OS::TripleO::Services::NeutronDhcpAgent
        - OS::TripleO::Services::NeutronL2gwAgent
        - OS::TripleO::Services::NeutronL2gwApi
        - OS::TripleO::Services::NeutronL3Agent
        - OS::TripleO::Services::NeutronLinuxbridgeAgent
        - OS::TripleO::Services::NeutronMetadataAgent
        - OS::TripleO::Services::NeutronML2FujitsuCfab
        - OS::TripleO::Services::NeutronML2FujitsuFossw
        - OS::TripleO::Services::NeutronOvsAgent
        - OS::TripleO::Services::NeutronVppAgent
        - OS::TripleO::Services::NeutronAgentsIBConfig
        - OS::TripleO::Services::NovaApi
        - OS::TripleO::Services::NovaConductor
        - OS::TripleO::Services::NovaIronic
        - OS::TripleO::Services::NovaMetadata
        - OS::TripleO::Services::NovaScheduler
        - OS::TripleO::Services::NovaVncProxy
        - OS::TripleO::Services::ContainersLogrotateCrond
        - OS::TripleO::Services::OctaviaApi
        - OS::TripleO::Services::OctaviaDeploymentConfig
        - OS::TripleO::Services::OctaviaHealthManager
        - OS::TripleO::Services::OctaviaHousekeeping
        - OS::TripleO::Services::OctaviaWorker
        - OS::TripleO::Services::OpenStackClients
        - OS::TripleO::Services::OVNDBs
        - OS::TripleO::Services::OVNController
        - OS::TripleO::Services::Pacemaker
        - OS::TripleO::Services::PankoApi
        - OS::TripleO::Services::PlacementApi
        - OS::TripleO::Services::OsloMessagingRpc
        - OS::TripleO::Services::OsloMessagingNotify
        - OS::TripleO::Services::Podman
        - OS::TripleO::Services::Rear
        - OS::TripleO::Services::Redis
        - OS::TripleO::Services::Rhsm
        - OS::TripleO::Services::Rsyslog
        - OS::TripleO::Services::RsyslogSidecar
        - OS::TripleO::Services::SaharaApi
        - OS::TripleO::Services::SaharaEngine
        - OS::TripleO::Services::Securetty
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Sshd
        - OS::TripleO::Services::SwiftProxy
        - OS::TripleO::Services::SwiftDispersion
        - OS::TripleO::Services::SwiftRingBuilder
        - OS::TripleO::Services::SwiftStorage
        - OS::TripleO::Services::Timesync
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
        - OS::TripleO::Services::Tuned
        - OS::TripleO::Services::Vpp
        - OS::TripleO::Services::Zaqar
    ###############################################################################
    # Role: ComputeHCIOvsDpdkSriov                                                #
    ###############################################################################
    - name: ComputeHCIOvsDpdkSriov
      description: |
        ComputeOvsDpdkSriov Node role hosting Ceph OSD too
      networks:
        InternalApi:
          subnet: internal_api_subnet
        Tenant:
          subnet: tenant_subnet
        Storage:
          subnet: storage_subnet
        StorageMgmt:
          subnet: storage_mgmt_subnet
      # CephOSD present so serial has to be 1
      update_serial: 1
      RoleParametersDefault:
        TunedProfileName: "cpu-partitioning"
        VhostuserSocketGroup: "hugetlbfs"
        NovaLibvirtRxQueueSize: 1024
        NovaLibvirtTxQueueSize: 1024
      ServicesDefault:
        - OS::TripleO::Services::Aide
        - OS::TripleO::Services::AuditD
        - OS::TripleO::Services::BootParams
        - OS::TripleO::Services::CACerts
        - OS::TripleO::Services::CephClient
        - OS::TripleO::Services::CephExternal
        - OS::TripleO::Services::CephOSD
        - OS::TripleO::Services::CertmongerUser
        - OS::TripleO::Services::Collectd
        - OS::TripleO::Services::ComputeCeilometerAgent
        - OS::TripleO::Services::ComputeNeutronCorePlugin
        - OS::TripleO::Services::ComputeNeutronL3Agent
        - OS::TripleO::Services::ComputeNeutronMetadataAgent
        - OS::TripleO::Services::ComputeNeutronOvsDpdk
        - OS::TripleO::Services::Docker
        - OS::TripleO::Services::IpaClient
        - OS::TripleO::Services::Ipsec
        - OS::TripleO::Services::Iscsid
        - OS::TripleO::Services::Kernel
        - OS::TripleO::Services::LoginDefs
        - OS::TripleO::Services::MetricsQdr
        - OS::TripleO::Services::Multipathd
        - OS::TripleO::Services::MySQLClient
        - OS::TripleO::Services::NeutronBgpVpnBagpipe
        - OS::TripleO::Services::NeutronSriovAgent
        - OS::TripleO::Services::NeutronSriovHostConfig
        - OS::TripleO::Services::NovaAZConfig
        - OS::TripleO::Services::NovaCompute
        - OS::TripleO::Services::NovaLibvirt
        - OS::TripleO::Services::NovaLibvirtGuests
        - OS::TripleO::Services::NovaMigrationTarget
        - OS::TripleO::Services::OvsDpdkNetcontrold
        - OS::TripleO::Services::ContainersLogrotateCrond
        - OS::TripleO::Services::Podman
        - OS::TripleO::Services::Rear
        - OS::TripleO::Services::Rhsm
        - OS::TripleO::Services::Rsyslog
        - OS::TripleO::Services::RsyslogSidecar
        - OS::TripleO::Services::Securetty
        - OS::TripleO::Services::Snmp
        - OS::TripleO::Services::Sshd
        - OS::TripleO::Services::Timesync
        - OS::TripleO::Services::Timezone
        - OS::TripleO::Services::TripleoFirewall
        - OS::TripleO::Services::TripleoPackages
        - OS::TripleO::Services::OVNController
        - OS::TripleO::Services::OVNMetadataAgent
        - OS::TripleO::Services::Ptp

14.2. network-environment-overrides.yaml

---
parameter_defaults:
  # The tunnel type for the tenant network (geneve or vlan). Set to '' to disable tunneling.
  NeutronTunnelTypes: "geneve"
  # The tenant network type for Neutron (vlan or geneve).
  NeutronNetworkType: ["geneve", "vlan"]
  NeutronExternalNetworkBridge: "'br-access'"
  # NTP server configuration.
  # NtpServer: ["clock.redhat.com"]
  # MTU global configuration
  NeutronGlobalPhysnetMtu: 9000
  # Configure the classname of the firewall driver to use for implementing security groups.
  NeutronOVSFirewallDriver: openvswitch
  SshServerOptionsOverrides:
    UseDns: "no"
  # Enable log level DEBUG for supported components
  Debug: true

  # From Rocky live migration with NumaTopologyFilter disabled by default
  # https://bugs.launchpad.net/nova/+bug/1289064
  NovaEnableNUMALiveMigration: true
  NeutronPluginExtensions: "port_security,qos,segments,trunk,placement"
  # RFE https://bugzilla.redhat.com/show_bug.cgi?id=1669584
  NeutronServicePlugins: "ovn-router,trunk,qos,placement"
  NeutronSriovAgentExtensions: "qos"

  ############################
  #  Scheduler configuration #
  ############################
  NovaSchedulerEnabledFilters:
    - AvailabilityZoneFilter
    - ComputeFilter
    - ComputeCapabilitiesFilter
    - ImagePropertiesFilter
    - ServerGroupAntiAffinityFilter
    - ServerGroupAffinityFilter
    - PciPassthroughFilter
    - NUMATopologyFilter
    - AggregateInstanceExtraSpecsFilter
  ComputeOvsDpdkSriovNetworkConfigTemplate: "/home/stack/ospd-17.0-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/nic-configs/computeovsdpdksriov.yaml"
  ControllerSriovNetworkConfigTemplate: "/home/stack/ospd-17.0-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/nic-configs/controller.yaml"

14.3. controller.j2

---
{% set mtu_list = [ctlplane_mtu] %}
{% for network in role_networks if network not in 'Tenant,External' %}
{{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
{%- endfor %}
{% set min_viable_mtu = mtu_list | max %}
network_config:
- type: interface
  name: nic1
  use_dhcp: false
  addresses:
  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }}
  routes:
  - ip_netmask: 169.254.169.254/32
    next_hop: {{ ctlplane_ip }}

- type: linux_bond
  name: bond_api
  mtu: {{ min_viable_mtu }}
  bonding_options: mode=active-backup
  use_dhcp: false
  dns_servers: {{ ctlplane_dns_nameservers }}
  members:
  - type: interface
    name: nic2
    primary: true

{% for network in role_networks if network not in 'Tenant,External' %}
- type: vlan
  mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
  device: bond_api
  vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
  addresses:
  - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
{% endfor %}

- type: ovs_bridge
  name: br-link0
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic3
    mtu: 9000
  - type: vlan
    vlan_id: {{ lookup('vars', networks_lower['Tenant'] ~ '_vlan_id') }}
    mtu: 9000
    addresses:
    - ip_netmask: {{ lookup('vars', networks_lower['Tenant'] ~ '_ip') }}/{{ lookup('vars', networks_lower['Tenant'] ~ '_cidr') }}

- type: ovs_bridge
  name: br-dpdk0
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic4
    mtu: 9000

- type: ovs_bridge
  name: br-dpdk1
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic5
    mtu: 9000

- type: ovs_bridge
  name: br-sriov1
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic6
    mtu: 9000

- type: ovs_bridge
  name: br-sriov2
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic7
    mtu: 9000

- type: interface
  name: nic8
  use_dhcp: false
  defroute: false

- type: interface
  name: nic9
  use_dhcp: false
  defroute: false

- type: ovs_bridge
  name: br-access
  use_dhcp: false
  mtu: 9000
  members:
  - type: interface
    name: nic10
    mtu: 9000
  - type: vlan
    vlan_id: {{ lookup('vars', networks_lower['External'] ~ '_vlan_id') }}
    mtu: 9000
    addresses:
    - ip_netmask: {{ lookup('vars', networks_lower['External'] ~ '_ip') }}/{{ lookup('vars', networks_lower['External'] ~ '_cidr') }}
    routes:
    - default: true
      next_hop: {{ lookup('vars', networks_lower['External'] ~ '_gateway_ip') }}

14.4. compute-ovs-dpdk.j2

---
{% set mtu_list = [ctlplane_mtu] %}
{% for network in role_networks if network not in 'Tenant,External' %}
{{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
{%- endfor %}
{% set min_viable_mtu = mtu_list | max %}
network_config:
- type: interface
  name: nic1
  use_dhcp: false
  default: no

- type: interface
  name: nic2
  use_dhcp: false
  addresses:
  - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }}
  routes:
  - ip_netmask: 169.254.169.254/32
    next_hop: {{ ctlplane_ip }}
  - default: true
    next_hop: {{ ctlplane_gateway_ip }}

- type: linux_bond
  name: bond_api
  mtu: {{ min_viable_mtu }}
  bonding_options: mode=active-backup
  use_dhcp: false
  dns_servers: {{ ctlplane_dns_nameservers }}
  members:
  - type: interface
    name: nic2
    primary: true

{% for network in role_networks if network not in 'Tenant,External' %}
- type: vlan
  mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
  device: bond_api
  vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
  addresses:
  - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
{% endfor %}

- type: ovs_user_bridge
  name: br-link0
  use_dhcp: false
  ovs_extra: "set port br-link0 tag={{ lookup('vars', networks_lower['Tenant'] ~ '_vlan_id') }}"
  addresses:
  - ip_netmask: {{ lookup('vars', networks_lower['Tenant'] ~ '_ip') }}/{{ lookup('vars', networks_lower['Tenant'] ~ '_cidr')}}
  members:
  - type: ovs_dpdk_bond
    name: dpdkbond0
    rx_queue: 1
    ovs_extra: "set port dpdkbond0 bond_mode=balance-slb"
    members:
      - type: ovs_dpdk_port
        name: dpdk0
        members:
          - type: interface
            name: nic7
      - type: ovs_dpdk_port
        name: dpdk1
        members:
          - type: interface
            name: nic8

- type: ovs_user_bridge
  name: br-dpdk0
  use_dhcp: false
  mtu: 9000
  rx_queue: 1
  members:
    - type: ovs_dpdk_port
      name: dpdk2
      members:
        - type: interface
          name: nic5

- type: ovs_user_bridge
  name: br-dpdk1
  use_dhcp: false
  mtu: 9000
  rx_queue: 1
  members:
    - type: ovs_dpdk_port
      name: dpdk3
      members:
        - type: interface
          name: nic6

- type: sriov_pf
  name: nic9
  mtu: 9000
  numvfs: 10
  use_dhcp: false
  defroute: false
  nm_controlled: true
  hotplug: true
  promisc: false

- type: sriov_pf
  name: nic10
  mtu: 9000
  numvfs: 10
  use_dhcp: false
  defroute: false
  nm_controlled: true
  hotplug: true
  promisc: false

14.5. overcloud_deploy.sh

#!/bin/bash

tht_path='/home/stack/ospd-17.0-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid'
[[ ! -d "$tht_path/roles" ]] && mkdir $tht_path/roles
openstack overcloud roles generate -o $tht_path/roles/roles_data.yaml ControllerSriov ComputeOvsDpdkSriov

openstack overcloud deploy \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --ntp-server clock.redhat.com,time1.google.com,time2.google.com,time3.google.com,time4.google.com \
  --stack overcloud \
  --roles-file $tht_path/roles/roles_data.yaml \
  -n $tht_path/network/network_data_v2.yaml \
  --deployed-server \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  -e /home/stack/templates/overcloud-networks-deployed.yaml \
  -e /home/stack/templates/overcloud-vip-deployed.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dpdk.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-sriov.yaml \
  -e /home/stack/containers-prepare-parameter.yaml \
  -e $tht_path/network-environment-overrides.yaml \
  -e $tht_path/api-policies.yaml \
  -e $tht_path/bridge-mappings.yaml \
  -e $tht_path/neutron-vlan-ranges.yaml \
  -e $tht_path/dpdk-config.yaml \
  -e $tht_path/sriov-config.yaml \
  --log-file overcloud_deployment.log