OVN/OVS offloading with OpenShift on NVIDIA BlueField-2 DPUs

Updated -

OpenShift Container Platform installed on DPUs facilitates OVN/OVS offloading.

Note
OVN/OVS offloading with OpenShift on NVIDIA BlueField-2 DPUs is a Developer Preview feature in OpenShift Container Platform 4.10 only. It is not available on previous versions of OpenShift Container Platform.

About Developer Preview features
Developer Preview features are not supported with Red Hat production service level agreements (SLAs) and are not functionally complete. Red Hat does not advise using them in a production setting. Developer Preview features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. These releases may not have any documentation, and testing is limited. Red Hat may provide ways to submit feedback on Developer Preview releases without an associated SLA.

The features described in this document are for Developer Preview purposes and are not supported by Red Hat at this time.

About OVN/OVS offloading with OpenShift on NVIDIA BlueField-2 DPUs

Learn how to accelerate and offload software subsystems using OpenShift Container Platform and the Data Processing Unit (DPU). DPUs are a class of reprogrammable high-performance processors combined with high-performance network interfaces optimized to perform and accelerate network and storage functions carried out by data center servers. This infrastructure element that may also referred to an Intelligence Processing Unit (IPI) by some hardware vendors takes the form of a server hosted PCIe add-in card containing one or more ASICs, usually anchored around a single powerful System on a chip (SoC) device.

The DPU is a complete compute system with an independent software stack, network identity, and provisioning capabilities. The DPU is fully capable of hosting its own applications using either embedded or orchestrated deployment models.

The unique capabilities of the DPU are disruptive because they allow for key infrastructure functions and their associated software stacks to be completely removed from the host node’s CPU cores and to be relocated onto the DPU.

OVN/OVS flow offload is only supported on OpenShift Container 4.10.

Installing and configuring an accelerated infrastructure with OpenShift and DPUs

Installing OpenShift Container Platform on a DPU makes it possible to offload packet processing from the host x86 to the DPU. Offloading resource-intensive computational tasks, such as packet processing, from the server’s CPU to the DPU frees up cycles cycles on the OpenShift Container Platform worker nodes to run the applications.

OpenShift and DPU deployment architecture

The proposed deployment architecture is a two-cluster design. In this architecture, DPU cards are provisioned as worker nodes of the ARM-based infrastructure cluster. The tenant cluster, composed of the x86 servers, is where the normal user applications run. The following diagram illustrates the deployment architecture.

OpenShift Container Platform on DPU

The steps involved in deploying and configuring are:

  1. Install assisted installer on the installer node using podman.

  2. Install the infrastructure cluster.

  3. Install the tenant cluster.

  4. Install the DPU Network Operator on the infrastructure cluster.

  5. Enable SR-IOV Operator on the hosts with DPU mode.

Assisted Installer overview

The assisted installer simplifies the OpenShift Container Platform installation for several different platforms, but focuses on bare metal deployments.

The high-level flow for the user is:

  1. Create a cluster in the assisted installer UI installed on the installer node.

  2. Generate a discovery ISO. The discovery ISO is a live Red Hat Enterprise Linux CoreOS (RHCOS) image that the nodes will boot from.

  3. Download a discovery ISO.

  4. Boot the hosts in the infrastructure cluster with that discovery ISO.

  5. When booted an introspection is performed by the discovery agent and data sent to the assisted service.

  6. The assisted installer interface guides the user through the OpenShift Container Platform install.

  7. When everything is ready the user starts the cluster installation on the infrastructure cluster.

  8. Follow the same procedure steps 2 to 7 and start the cluster installation on the tenant cluster.

Install Assisted Installer on the installer node using Podman

Use podman to install the assisted installer on your installer node.

  • You have podman version 3.3+ installed.
  1. Download pod.yml.

    $ wget https://raw.githubusercontent.com/rh-ecosystem-edge/ansible-ai-playbook/main/examples/assisted-installer/pod.yml
    
  2. Download configmap.yml.

    $ wget https://raw.githubusercontent.com/rh-ecosystem-edge/ansible-ai-playbook/main/examples/assisted-installer/configmap.yml
    
  3. Create the directory /home/assisted-podman/pg_data

    $ mkdir -p /home/assisted-podman/pg_data
    
  4. Update IMAGE_SERVICE_BASE_URL and SERVICE_BASE_URL in configmap.yml updating the < HOST IP > to match the hostname or IP address of your host.

    For example if your IP address is 192.168.122.2, set the SERVICE_BASE_URL to http://192.168.122.2:8090

    Port 8090 is the assisted-service API that agents connect to.

  5. Use the following command to install Assisted Installer by using podman.

    $ podman play kube --configmap configmap.yml pod.yml
    
  6. Open a browser and access the assisted installer UI on:

    http://<host-ip-address>:8080
    

Installing the infrastructure cluster

Use the assisted installer to install OpenShift Container Platform on the infrastructure cluster running on ARM architecture.

  • Shared VLAN: The assisted installer requires all nodes to be present on the same VLAN. This is a requirement of utilizing a virtual IP for both ingress and API.

  • DHCP: Much like an installer-provisioned infrastructure (IPI) installation DHCP is required in the VLAN.

  • DNS: Records for ingress and API are required for accessing the cluster.

Generating the discovery ISO with the Assisted Installer for the infrastructure nodes

Installing OpenShift Container Platform infrastructure cluster (3 Control plane nodes) requires a discovery ISO, which the assisted installer (AI) can generate with the cluster name, base domain, Secure Shell (SSH) public key, and pull secret.

  1. Open a browser and access the Assisted Installer UI on:

    http://<host-ip-address>:8080
    
  2. Click Create Cluster to create a new cluster.

  3. In the Cluster name field, enter a name for the cluster.

  4. In the Base domain field, enter a base domain. For example:

    example.com
    

    All DNS records must be subdomains of this base domain and include the cluster name. You cannot change the base domain after cluster installation. For example:

    <cluster-name>.example.com
    
  5. Select the OpenShift Container Platform version.

  6. Tick Use arm64 CPU architecture.

  7. Paste in your pull secret. The pull secret is obtained from cloud.redhat.com.

  8. The Enable encryption of installation disks on control plane nodes and Enable encryption of installation disks on workers can remain at the default setting.

  9. Click Next.

    The process moves on to step 2 Host discovery

  10. Click Add hosts

  11. Select Full image file to boot with a USB drive or PXE. Select Minimal image file to boot with virtual media.

  12. Add the SSH public key of the installer node to the Public key field.

  13. Click Generate Discovery ISO.

  14. Click Download the discovery ISO.

    The option exists to copy the wget command. You may prefer to get the ISO by using the wget command.

Booting machines with the discovery ISO

To install OpenShift Container Platform, you must ensure the hosts boot from the discovery ISO. You can install from the ISO image burnt to disk or USB media or by using network PXE booting.

What method to use depends on your situation. A PXE install requires an available DHCP service and more preparation, but can make the installation process more automated. An install from the ISO burnt to USB media for example is a more manual process and can be inconvenient if you are setting up more than a few machines.

Installing using PXE or iPXE booting

You can use PXE or iPXE booting to install the discovery ISO on the machines in the cluster.

  1. Reboot each host in the cluster in turn server and intercept the boot sequence (often by hitting the F12 key on an attached keyboard) entering the BIOS settings.

  2. Change the boot order to boot from On-board Network Device (PXE) first.

  3. Input the location of the discovery ISO.

  4. Save and exit the BIOS settings. The server will boot with the discovery image.

  5. Monitor the progress of the installation of the discovery ISO on the console of the machine.

Installing with USB media

Installing with USB media involves creating a bootable USB drive with the discovery ISO on the installer node.

  1. On the installer node, insert a USB drive into a USB port.

  2. Create a bootable USB drive:

    # dd if=<path-to-iso> of=<path/to/usb> status=progress
    

    For example:

    # dd if=discovery_image_sno.iso of=/dev/sdb status=progress
    

    After the ISO is copied to the USB drive, you can use the USB drive to install OpenShift Container Platform.

  3. On the server, insert the USB drive into a USB port.

  4. Reboot the server and enter the BIOS settings upon reboot.

  5. Change boot drive order to make the USB drive boot first.

  6. Save and exit the BIOS settings. The server will boot with the discovery image.

Monitoring the installation with the Assisted Installer

Use this procedure to monitor the installation in the infrastructure cluster.

  1. On the installer node, return to the browser and refresh the page.

  2. Select the cluster name.

  3. Click Next to reach the Host discovery screen. It should now be populated with your three control plane hosts.

  4. Click Next until you reach step 3, Networking.

  5. Choose Cluster-Managed Networking.

  6. Select a subnet from the available subnets.

  7. Create an API Virtual IP. This IP must be in the available subnet range.

  8. Create an Ingress Virtual IP. This IP must be in the available subnet range.

  9. Set Network type to Open Virtual Networking

  10. Keep Use the same host discovery SSH key checked. You can change the SSH public key, if necessary.

  11. Click Next to proceed to the Review and create step.

  12. Click Install cluster. The install completes in about 30 to 40 minutes.

  13. Monitor the installation’s progress. Watch the cluster events. After the installation process finishes writing the discovery image to the servers drive, the servers in the cluster will restart. If you installed using USB drive or PXE boot reset the BIOS on the servers in the cluster to boot to the servers' local media rather than the USB drive.

The servers will restart several times, deploying the control plane.

Download and extract the discovery ISO using ansible

The DPU network card does not currently allow you to install using virtual media so you need to use the PXE boot method. An ansible repo is available to ensure the download and extraction of the discovery ISO needed to PXE boot these additional hosts is easy to perform. The playbooks in this repo are used to:

  • Add worker nodes to the existing infrastructure cluster.

  • Transform the cluster into Day2.

  • Set up the PXE services putting the files in the correct location to boot the added hosts.

The Ansible Assisted Installer Playbooks includes two different roles:

  • setup-pxe: This role installs the basic services required for PXE booting namely:

    • HTTP server

    • TFTP server

    • Optional: DHCP server. Add the configuration for DHCP to the playbook.yaml to install the DHCP service.

      DHCP is optional as you may already have DHCP setup.

  • download-iso-pxe: This role is for Assisted Installer day2 deployments. Giving an existing cluster it adds the worker node using the API. This role also transforms the cluster to Day2, downloads and extracts the ISO into HTTP and TFTP folders. In addition it generates the grub.cfg.

This procedure describes how to download and install the discovery ISO. Hosts have already been added to the existing cluster.

  • You have ansible version 2.9 or greater installed.

  • You have installed jq.See the jq Manual for detailed information about using jq.

  1. Clone the Ansible Assisted Installer Playbooks repo.

    Clone the repo on the installer node or on your laptop (that has access to the installer node).

    $ git clone https://github.com/rh-ecosystem-edge/ansible-ai-playbook.git
    
  2. Edit the playbook.yaml.

    ---
    - hosts: all
      roles:
        - setup-pxe
        - download-iso-pxe
      vars:
        ARCH: arm
        URL: http://<URL>:<PORT>
        CLUSTER_ID: "<CLUSTER UUID>"
    
    • setup-pxe: This field sets up the basic services required for PXE booting

    • URL: Enter the Assisted Installer URL and port number

    • CLUSTER_ID: Enter the cluster UUID from the Assisted Installer UI.

    The option exists to download the ISO manually and extract what is needed to support PXE booting the hosts. To do this add two variables to the playbook.yaml. The variables are:

    • ISO_NAME: "discovery_image.iso" - This variable is the name of the downloaded ISO.

    • WORKDIR: "/tmp/download-iso-pxe"- This variable specifies the directory where the ISO is downloaded to.

  3. Download and extract the discovery ISO using the following command.

    $ ansible-playbook -i localhost, playbook.yaml --tags extact,download
    

    Use localhost if running the command on the installer node. Replace localhost with the FQDN of the installer node if running the command on your laptop. By default the name of the downloaded ISO is discovery_image.iso and it is downloaded to /tmp/download-iso-pxe.

  4. Verify the discovery image exists in /tmp/download-iso-pxe.

    1. List the contents of the download directory.

      $ ll /tmp/download-iso-pxe
      

      Expected output

      discovery_image.iso
      
  5. Verify the correct files are extracted to the correct locations.

    1. Verify the ignition file and rootfs image exists in /var/www/html/pxe.

      $ ll /var/www/html/pxe
      

      Expected output

      total 839720
      -rw-r--r--. 1 apache apache     10484 Mar 23 06:15 config.ign
      -rw-r--r--. 1 apache apache 859858944 Mar 23 06:15 rootfs.img
      
    2. Verify the grub.cfg and the files needed to boot the VM exist in /var/lib/tftpboot.

      $ ll /var/lib/tftpboot
      

      Expected output

      total 89956
      -rw-r--r--. 1 root root   857984 Jan 28 01:13 BOOTAA64.EFI
      -rw-r--r--. 1 root root  2446280 Jan 28 01:13 grubaa64.efi
      -rw-r--r--. 1 root root      567 Mar 23 06:16 grub.cfg
      -rw-r--r--. 1 root root 79229244 Mar 23 06:06 initrd.img
      -rw-r--r--. 1 root root  9565104 Mar 23 06:06 vmlinuz
      

At this stage you have all the files required to PXE boot the DPU hosts.

PXE boot additional hosts with the discovery ISO

Use PXE to boot the DPU nodes in the cluster using the discovery ISO. There are two possible ways to do this:

  1. Interrupt the boot sequence and enter the BIOS settings

  2. Configure DHCP to store and serve the boot information

  1. Interrupt the boot sequence and enter the BIOS settings.

    1. Reboot each host in the cluster in turn by intercepting the boot sequence (often by hitting the F12 key on an attached keyboard) entering the BIOS settings.

    2. Change the boot order to boot from On-board Network Device (PXE) first.

    3. Input the location of the discovery ISO.

    4. Save and exit the BIOS settings. The server will boot with the discovery image.

    5. Monitor the progress of the installation of the discovery ISO on the console of the machine.

    6. Ensure you set the boot order back to Boot from disk before starting the installation.

  2. Configure DHCP to store and serve the boot information.

    1. Edit /etc/dhcp/dhcpd.conf and append an entry for the PXE clients, for example:

        class "pxeclients" {
              match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
              next-server 172.31.100.1;
              always-broadcast on;
      
              filename "/BOOTAA64.EFI";
      
      • next-server: Specify the IP address of the TFTP server.

      • filename: Specify the EFI program file.

    2. Restart the DHCP server

      $ systemctl restart dhcpd.service
      
    3. Change the boot order to boot from PXE using command line:

      $ ipmitool -H {{ BMC_ADDRESS }} -U {{ BMC_USER }} -P {{ BMC_PASSWORD }}  -I lanplus chassis bootdev pxe
      
    4. Inititiate a power reset:

      $ ipmitool -H {{ BMC_ADDRESS }} -U {{ BMC_USER }} -P {{ BMC_PASSWORD }}  -I lanplus power reset
      
    5. After booting from ISO revert the boot order back to disk:

      $ ipmitool -H {{ BMC_ADDRESS }} -U {{ BMC_USER }} -P {{ BMC_PASSWORD }}  -I lanplus chassis bootdev disk
      

Installing OpenShift Container Platform on the DPU hosts

Install OpenShift Container Platform on the DPU hosts using this procedure carried out on the installer node.

The example output reflects the installation on one DPU node.

  1. Export the API URL environment variable:

    $ export API_URL=http://<Installer_FQDN>:8080/api/assisted-install/v2
    
  2. Export the CLUSTER_UUID environment variable:

    $ export CLUSTER_ID=<CLUSTER_UUID>
    
  3. Get the INFRAENV id.

    $ curl --silent "$API_URL/infra-envs?cluster_id=$CLUSTER_ID" | jq '.[] .id'
    

    Example output

    aff04425-8f9f-44bb-8c4b-b83d689a10a4
    
  4. Export the INFRAENV environment variable:

    $ export INFRAENV=aff04425-8f9f-44bb-8c4b-b83d689a10a4
    
    1. Get all host and hostnames registered to the INFRAENV.

      $ curl --silent "$API_URL/infra-envs/$INFRAENV/hosts" | jq '.[] |  { id: .id, requested_hostname: .requested_hostname}'
      

      Example output

      {
        "id": "acfa0fe6-6399-cff8-d344-fe635c3c029c",
        "requested_hostname": "localhost"
      }
      {
        "id": "9fd88bba-bd3d-44f0-9399-3750c31922b9",
        "requested_hostname": "master-3"
      }
      {
        "id": "a7823fea-9f80-4c11-aefb-ad6e96b1b4b2",
        "requested_hostname": "master-1"
      }
      {
        "id": "2b461591-27dc-4531-9a9b-ee33e98ecd6f",
        "requested_hostname": "master-2"
      }
      

      You cannot install with hostname set to localhost. This needs to be changed.

    2. Set the localhost hostname to the id:

      $ curl --silent -X PATCH   "$API_URL/infra-envs/$INFRAENV/hosts/$HOST_ID"   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '{\n "host-role": "worker" "host_name": "worker-bf" }'
      
    3. Verify the hostname is set:

      $ curl --silent "$API_URL/infra-envs/$INFRAENV/hosts" | jq '.[] |  { id: .id, requested_hostname: .requested_hostname}'
      

      Example output

      {
        "id": "acfa0fe6-6399-cff8-d344-fe635c3c029c",
        "requested_hostname": "worker-bf"
      }
      {
        "id": "9fd88bba-bd3d-44f0-9399-3750c31922b9",
        "requested_hostname": "master-3"
      }
      {
        "id": "a7823fea-9f80-4c11-aefb-ad6e96b1b4b2",
        "requested_hostname": "master-1"
      }
      {
        "id": "2b461591-27dc-4531-9a9b-ee33e98ecd6f",
        "requested_hostname": "master-2"
      }
      
  5. Start the installation.

    1. Set the HOST_ID:

      $ HOST_ID=acfa0fe6-6399-cff8-d344-fe635c3c029c
      
    2. Make a POST request to the API using curl to start the installation:

      $ curl -X POST --silent "$API_URL/infra-envs/$INFRAENV/hosts/$HOST_ID/actions/install" | jq .
      
  6. Verify the installation status.

    1. Set the HOST_ID:

      $ HOST_ID=acfa0fe6-6399-cff8-d344-fe635c3c029c
      
    2. Use curl to verify the status of the install:

      $ curl --silent "$API_URL/infra-envs/$INFRAENV/hosts/$HOST_ID" | jq '.status'
      
    3. Verify the progress of the installation:

      $ curl --silent "$API_URL/infra-envs/$INFRAENV/hosts/$HOST_ID" | jq '.progress'
      
  7. Approve the certificate request.

    Two certificate requests are generated and both need approved. The first one generated is the kube-apiserver-client-kubelet certificate request. Once this is approved a second certificate request the kubelet-serving is generated.

    1. Get the certificate request:

      $ oc get csr
      
    2. Approve the kube-apiserver-client-kubelet certificate request:

      $ oc adm certificate approve <csr_name>
      
    3. Approve the kubelet-serving certificate request:

      $ oc adm certificate approve <csr_name>
      
    4. Verify that both certificates are issued and approved:

      $ oc get csr
      
  8. Verify the DPU node now exists:

    $ oc get csr
    

    Example output

    NAME                      STATUS    ROLES            AGE       VERSION
    worker-bf                 Ready     worker           35s        v1.23.3+bba255d
    master-0                  Ready     master,worker    108m       v1.23.3+bba255d
    master-1                  Ready     master,worker    108m       v1.23.3+bba255d
    master-2                  Ready     master,worker    89m        v1.23.3+bba255d
    

OpenShift Container Platform is now succesfully installed on the DPU node.

  1. Post installation the Assisted Installer UI is broken. Resolve as follows.

    1. Connect to the Assisted Installer database:

      $ podman exec -ti assisted-installer-db bash
      
    2. Run the following psql command:

      $ psql installer admin
      
    3. Update the Assisted Installer cluster status with the following command:

      installer => update clusters set kind='Cluster', status='installed' where id='<CLUSTER UUID>';
      

Installing the tenant cluster

Use the assisted installer to install the tenant cluster.

  • Shared VLAN: The assisted installer requires all nodes to be present on the same VLAN. This is a requirement of utilizing a virtual IP for both ingress and API.

  • DHCP: Much like an IPI installation DHCP is required in the VLAN.

  • DNS: Records for ingress and API are required for accessing the cluster.

Generating the discovery ISO with the Assisted Installer

Installing OpenShift Container Platform x86 tenant cluster (3 Control plane nodes and 3 workers nodes) requires a discovery ISO, which the assisted installer (AI) can generate with the cluster name, base domain, Secure Shell (SSH) public key, and pull secret.

  1. Open a browser and access the assisted installer UI on:

    http://<host-ip-address>:8080
    
  2. Click Create Cluster to create a new cluster.

  3. In the Cluster name field, enter a name for the cluster.

  4. In the Base domain field, enter a base domain. For example:

    example.com
    

    All DNS records must be subdomains of this base domain and include the cluster name. You cannot change the base domain after cluster installation. For example:

    <cluster-name>.example.com
    
  5. Select the OpenShift Container Platform version.

  6. Paste in your pull secret. The pull secret can be obtained from cloud.redhat.com.

  7. Optional: Toggle on Enable encryption of installation disks on control plane nodes if you want to encrypt installation disks on the control plane nodes with TPM v2 or Tang encryption.

  8. Optional: Toggle on Enable encryption of installation disks on workers if you want to encrypt installation disks on the workers nodes by using TPM v2 or Tang encryption.

  9. Click Next.

    The process moves on to step 2 Host discovery

  10. Click Add hosts

  11. Select Full image file to boot with a USB drive or PXE. Select Minimal image file to boot with virtual media.

  12. Add SSH public key of the installer node to the Public key field.

  13. Click Generate Discovery ISO.

  14. Click Download the discovery ISO.

    Make a note of the discovery ISO URL for installing with virtual media.

Booting machines with the discovery ISO

To install OpenShift Container Platform, you must ensure the hosts boot from the discovery ISO. You can install from the ISO image burnt to disk or USB media or by using network PXE booting.

What method to use depends on your situation. A PXE install requires an available DHCP service and more preparation, but can make the installation process more automated. An install from the ISO burnt to USB media for example is a more manual process and can be inconvenient if you are setting up more than a few machines.

Monitoring the installation with the Assisted Installer

Use this procedure to monitor the installation and assign roles to the nodes in the tenant cluster.

  1. On the installer node, return to the browser and refresh the page.

  2. Select the cluster name.

  3. Click Next to reach the Host discovery screen. It should now be populated with your cluster hosts.

  4. Under Role assign 3 control plane nodes and 3 worker nodes.

  5. Click Next until you reach step 3, Networking.

  6. Choose Cluster-Managed Networking.

  7. Select a subnet from the available subnets.

  8. Create an API Virtual IP. This IP must be in the available subnet range.

  9. Create an Ingress Virtual IP. This IP must be in the available subnet range.

  10. Set Network type to Open Virtual Networking.

  11. Keep Use the same host discovery SSH key checked. You can change the SSH public key, if necessary.

  12. Click Next to proceed to the Review and create step.

  13. Click Install cluster.

  14. Monitor the installation’s progress. Watch the cluster events. After the installation process finishes writing the discovery image to the servers drive, the servers in the cluster will restart. If you installed by using USB drive or PXE boot reset the BIOS on the servers in the cluster to boot to the servers' local media rather than the USB drive.

The servers will restart several times, deploying the control plane.

Installing the DPU Network Operator

The DPU Network Operator on the infrastructure cluster side is responsible for the life-cycle management of the ovn-kube components and the necessary host network initialization on DPU cards.

As a cluster administrator, you can install the DPU network Operator using the OpenShift Container Platform CLI or the web console.

  • An infrastructure cluster composed of ARM hosts is up and running.

  • An tenant cluster composed of x86 hosts.

  • DPU cards installed on the worker nodes of the infrastructure cluster where hardware offloading need to be enabled.

  • Pods in infrastructure cluster can reach the API server of tenant cluster

  • Network configuration:

    • The infrastructure cluster and the tenant cluster share the same VLAN as the API network.

    • The two clusters shall use different VIPs as the cluster API VIP.

    • The DHCP server needs to be configured to be able to assign IP addresses to hosts from both clusters from the same subnet.

    • The DNS server shall be able to resolve the URI to the API URL for both the clusters.

Installing the Operator using the web console

As a cluster administrator, you can install the DPU Network Operator using the web console.

  1. Install the DPU Network Operator using the OpenShift Container Platform web console (on the infrastructure cluster):

    1. In the OpenShift Container Platform web console, click OperatorsOperatorHub.

    2. Choose DPU Network Operator from the list of available Operators, and then click Install.

    3. On the Install Operator page, the only installation mode supported is All namespaces on the cluster. Review settings and click Install.

  2. Optional: Verify that the DPU Network Operator installed successfully:

    1. Switch to the OperatorsInstalled Operators page.

    2. Ensure that DPU Network Operator is listed in the openshift-operators project with a Status of InstallSucceeded.

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the OperatorsInstalled Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors
        under Status.

      • Go to the WorkloadsPods page and check the logs for the dpu-network-operator-controller-manager pod.

Installing the Operator using the CLI

As a cluster administrator, you can install the Operator using the CLI.

  • Install the OpenShift CLI (oc).

  • Log in to infrastructure cluster as a user with cluster-admin privileges.

  1. Create the following Subscription CR and save the YAML in the dpu-sub.yaml file:

    Example Subscription

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: openshift-dpu-operator-subscription
      namespace: openshift-operators
    spec:
      channel: "stable"
      name: dpu-network-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    
    • source: You must specify the redhat-operators value.
  2. Create the Subscription object by running the following command:

    $ oc create -f dpu-sub.yaml
    
  3. Change to the openshift-operators project:

    $ oc project openshift-operators
    
  4. Verify the DPU Network Operator is running:

    $ oc get pods
    

    Example output

    NAME                                                      READY   STATUS    RESTARTS   AGE
    dpu-network-operator-controller-manager-cc9ccc4bd-9vqcg   2/2     Running   0          62s
    

Enabling hardware offloading with the DPU Network Operator

Enable hardware offloading in the deployed infrastructure and tenant cluster by using this procedure.

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Configuring support for hardware offloading in the infrastructure cluster

Configure support for hardware offloading in the infrastructure cluster by using this procedure.

  1. Create the following OVNKubeConfig Custom Resource (CR) with the poolName dpu, leaving the kubeConfigFile blank. Save the YAML in the ovnkubeconfig.yaml file:

    apiVersion: dpu.openshift.io/v1alpha1
    kind: OVNKubeConfig
    metadata:
      name: ovnkubeconfig-sample
    spec:
      poolName: dpu
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/dpu-worker: ""
    
    1. Run the following command:

      $ oc create -f ovnkubeconfig.yaml
      

      The DPU Network Operator creates a custom MachineConfigPool and a custom MachineConfig.

  2. Get the node names by using the following command:

    $ oc get nodes
    
  3. Label the DPU nodes in the infrastructure cluster. The Machine Config Operator applies the new MachineConfig to the DPU nodes, therefore enabling switchdev mode on them:

    $ oc label node <NODENAME> node-role.kubernetes.io/dpu-worker=
    
  4. Create a Cluster Network Operator (CNO) ConfigMap in the infrastructure cluster setting the mode to DPU. Save the YAML in the ovnkubeconfigmap.yaml file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
        name: dpu-mode-config
        namespace: openshift-network-operator
    data:
        mode: "dpu"
    immutable: true
    
    1. Run the following command:

      $ oc create -f ovnkubeconfigmap.yaml
      
  5. Label the DPU nodes in the infrastructure cluster where you want to enable hardware offloading.

    $ oc label node <NODENAME> network.operator.openshift.io/dpu=
    

Configuring support for hardware offloading in the tenant cluster

Configure support for hardware offloading in the tenant cluster by using this procedure.

  1. Create a MachineConfigPool for all the DPU workers. Save the YAML in the dputenantmachineconfig.yaml file:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfigPool
    metadata:
      name: dpu-host
    spec:
      machineConfigSelector:
        matchExpressions:
        - key: machineconfiguration.openshift.io/role
          operator: In
          values:
          - worker
          - dpu-host
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/dpu-host: ""
    
    1. Run the following command:

      $ oc create -f dputenantmachineconfig.yaml
      
  2. Label the DPU nodes:

    $ oc label node <NODENAME> node-role.kubernetes.io/dpu-host=
    
  3. Install and configure the SR-IOV Network Operator:

    1. In the OpenShift Container Platform web console, click AdministrationNamespaces.

    2. Click Create Namespace.

    3. In the Name field, enter openshift-sriov-network-operator, and click Create.

    4. In the OpenShift Container Platform web console, click OperatorsOperatorHub.

    5. Select SR-IOV Network Operator from the list of available Operators, and click Install.

    6. On the Install Operator page, under A specific namespace on the cluster, select openshift-sriov-network-operator.

    7. Click Install.

    8. Verify that the SR-IOV Network Operator is installed successfully:

      1. Navigate to the OperatorsInstalled Operators page.

      2. Ensure that SR-IOV Network Operator is listed in the openshift-sriov-network-operator project with a Status of InstallSucceeded.

        During installation an Operator might display a Failed status.
        If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

        If the operator does not appear as installed, to troubleshoot further:

        • Inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.

        • Navigate to the WorkloadsPods page and check the logs for pods in the
          openshift-sriov-network-operator project.

  4. Add this machine config pool to the SriovNetworkPoolConfig custom resource.

    1. Create a file, such as sriov-pool-config.yaml, with the following content:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkPoolConfig
      metadata:
        name: default
        namespace: openshift-sriov-network-operator
      spec:
        ovsHardwareOffloadConfig:
          name: dpu-host
      
      • name (under spec): The name here is the same as the machine config pool (MCP) name created in step 1.
    2. Apply the configuration:

      $ oc create -f sriov-pool-config.yaml
      
  5. Create a SriovNetworkNodePolicy to configure the virtual functions (VFs) on the hosts.

    1. Save the YAML in the SriovNetworkNodePolicy.yaml file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
        name: policy-mlnx-bf
        namespace: openshift-sriov-network-operator
      spec:
        resourceName: mlnx_bf
        nodeSelector:
          node-role.kubernetes.io/dpu-host: ""
        priority: 99
        numVfs: 4
        nicSelector:
          vendor: "15b3"
          deviceId: "a2d6"
          pfNames: ['ens1f0']
          rootDevices: ['0000:3b:00.0']
      
      • name: The name for the custom resource object.

      • resourceName: The resource name of the SR-IOV network device plug-in. You can create multiple SR-IOV network node policies for a resource name.

      • nodeSelector: The node selector specifies the nodes to configure. Ensure this is consistent with the nodeSelector of the MCP created in step 1.

      • priority: The optional priority value is an integer value between 0 and 99. A smaller value receives higher priority. For example, a priority of 10 is a higher priority than 99. The default value is 99.

      • numVfs: The number of the virtual functions (VF) to create for the SR-IOV physical network device. For a Mellanox NIC, the number of VFs cannot be larger than 128.

      • nicSelector: The NIC selector identifies the device for the Operator to configure. You do not have to specify values for all the parameters. It is recommended to identify the network device with enough precision to avoid selecting a device unintentionally.

      • vendor: The vendor hexadecimal code of the SR-IOV network device. Vendor id 15b3 is for Mellanox devices.

      • deviceId: The device hexadecimal code of the SR-IOV network device. For example, a2d6 is the device ID for a Bluefield-2 DPU device.

      • pfNames: An array of one or more physical function (PF) names for the device.

      • rootDevices: An array of one or more PCI bus addresses for the PF of the device. Provide the address in the following format: 0000:02:00.1.

    2. Create the SriovNetworkNodePolicy object:

      $ oc create -f SriovNetworkNodePolicy.yaml
      
  6. Create a Cluster Network Operator (CNO) ConfigMap in the tenant cluster setting the mode to dpu-host.

    1. Save the YAML in the sriovdpuconfigmap.yaml file:

      apiVersion: v1
      kind: ConfigMap
      metadata:
          name: dpu-mode-config
          namespace: openshift-network-operator
      data:
          mode: "dpu-host"
      immutable: true
      
    2. Run the following command:

      $ oc create -f sriovdpuconfigmap.yaml
      
  7. Set the environment variable OVNKUBE_NODE_MGMT_PORT_NETDEV for each DPU host.

    1. Save the YAML in the setenvovnkube.yaml file:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: env-overrides
        namespace: openshift-ovn-kubernetes
      data:
        x86-worker-node0: |
          OVNKUBE_NODE_MGMT_PORT_NETDEV=ens1f0v0
      
      • ens1f0v0 is the virtual function (VF) name that is assigned to the ovnkube node management port on the host.
    2. Run the following command:

      $ oc create -f setenvovnkube.yaml
      
  8. Label the DPU nodes in the tenant cluster. Run the following command:

    $ oc label node <NODENAME> network.operator.openshift.io/dpu-host=
    
  9. Label the DPU nodes in the tenant cluster with node-role. Run the following command:

    $ oc label node <NODENAME> node-role.kubernetes.io/dpu-host=
    

Final infrastructure cluster configuration

Run the following steps against the infrastructure cluster.

  1. Add the kubeconfig of the tenant cluster as a secret:

    $ oc create secret generic tenant-cluster-1-kubeconf --from-file=config=/root/kubeconfig.tenant
    
  2. Add the per node configuration override for the ovnkube-node by listing all the DPU nodes under :data:

    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: env-overrides
      namespace: default
    data:
      worker-bf: |
        TENANT_K8S_NODE=x86-worker-node0
        DPU_IP=192.168.111.29
        MGMT_IFNAME=pf0vf0
    
  3. Update the OvnkubeConfig CR by adding the kubeConfigFile field. Wait until the ovnkube pods are created.

    apiVersion: dpu.openshift.io/v1alpha1
    kind: OVNKubeConfig
    metadata:
      name: ovnkubeconfig-sample
    spec:
      kubeConfigFile: tenant-cluster-1-kubeconf
      poolName: dpu
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/dpu-worker: ''
    

In the DPU Network Operator there are several status messages that can be used to verify the Operator’s working status:

  • McpReady indicates that the MachineConfig and MachineConfigPool are ready.

  • TenantObjsSynced indicates that the tenant ConfigMap and secrets are synced to the infrastructure nodes.

  • OvnKubeReady indicates that the ovnkube-node DaemonSet is ready.