Procedure

1. Open the AWS console, fetch the master instances.
2. Stop one master instance.
3. Select the stopped instance, and click Actions → Instance Settings → Change instance type.
4. Change the instance to a larger type, ensuring that the type is the same base as the previous selection, and apply changes. For example, you can change m5.xlarge to m5.2xlarge or m5.4xlarge.
5. Backup the instance, and repeat the steps for the next master instance.

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New with Dialog.
3. Specify a unique name for the new bare metal host.
4. Set the Boot MAC address.
5. Set the Baseboard Management Console (BMC) Address.
6. Optionally, enable power management for the host. This allows OpenShift Container Platform to control the power state of the host.
7. Enter the user credentials for the host’s baseboard management controller (BMC).
8. Select to power on the host after creation, and select Create.
9. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machine replicas in the cluster by selecting Edit Machine count from the Actions drop-down menu.

Procedure

1. In the web console, navigate to Compute → Bare Metal Hosts.
2. Select Add Host → New from YAML.

3. Copy and paste the below YAML, modifying the relevant fields with the details of your host:

   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: <bare_metal_host_name>
   spec:
     online: true
     bmc:
       address: <bmc_address>
       credentialsName: <secret_credentials_name>  1
       disableCertificateVerification: True
     bootMACAddress: <host_boot_mac_address>
     hardwareProfile: unknown

   1

   credentialsName must reference a valid Secret CR. The baremetal-operator cannot manage the bare metal host without a valid Secret referenced in the credentialsName. For more information about secrets and how to create them, see Understanding secrets.

4. Select Create to save the YAML and create the new bare metal host.

5. Scale up the number of replicas to match the number of available bare metal hosts. Navigate to Compute → MachineSets, and increase the number of machines in the cluster by selecting Edit Machine count from the Actions drop-down menu.

   Note

   You can also manage the number of bare metal nodes using the oc scale command and the appropriate bare metal machine set.

Procedure

1. Annotate the machine set that you want to configure for automatic scaling by adding the metal3.io/autoscale-to-hosts annotation. Replace <machineset> with the name of the machine set.

   $ oc annotate machineset <machineset> -n openshift-machine-api 'metal3.io/autoscale-to-hosts=<any_value>'

   Wait for the new scaled machines to start.

1. Label all nodes that need the same huge pages setting by a label.

   $ oc label node <node_using_hugepages> node-role.kubernetes.io/worker-hp=

2. Create a file with the following content and name it hugepages-tuned-boottime.yaml:

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: hugepages 1
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile: 2
     - data: |
         [main]
         summary=Boot time configuration for hugepages
         include=openshift-node
         [bootloader]
         cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50 3
       name: openshift-node-hugepages
   
     recommend:
     - machineConfigLabels: 4
         machineconfiguration.openshift.io/role: "worker-hp"
       priority: 30
       profile: openshift-node-hugepages

   1

   Set the name of the Tuned resource to hugepages.

   2

   Set the profile section to allocate huge pages.

   3

   Note the order of parameters is important as some platforms support huge pages of various sizes.

   4

   Enable machine config pool based matching.

3. Create the Tuned hugepages profile

   $ oc create -f hugepages-tuned-boottime.yaml

4. Create a file with the following content and name it hugepages-mcp.yaml:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-hp
     labels:
       worker-hp: ""
   spec:
     machineConfigSelector:
       matchExpressions:
         - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-hp: ""

5. Create the machine config pool:

   $ oc create -f hugepages-mcp.yaml

Procedure

1. Create a namespace for the Performance Addon Operator by completing the following actions:

   1. Create the following Namespace Custom Resource (CR) that defines the openshift-performance-addon-operator namespace, and then save the YAML in the pao-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-performance-addon-operator

   2. Create the namespace by running the following command:

      $ oc create -f pao-namespace.yaml

2. Install the Performance Addon Operator in the namespace you created in the previous step by creating the following objects:

   1. Create the following OperatorGroup CR and save the YAML in the pao-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-performance-addon-operator
        namespace: openshift-performance-addon-operator
      spec:
        targetNamespaces:
        - openshift-performance-addon-operator

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f pao-operatorgroup.yaml

   3. Run the following command to get the channel value required for the next step.

      $ oc get packagemanifest performance-addon-operator -n openshift-marketplace -o jsonpath='{.status.defaultChannel}'

      Example output

      4.6

   4. Create the following Subscription CR and save the YAML in the pao-sub.yaml file:

      Example Subscription

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-performance-addon-operator-subscription
        namespace: openshift-performance-addon-operator
      spec:
        channel: "<channel>" 1
        name: performance-addon-operator
        source: redhat-operators 2
        sourceNamespace: openshift-marketplace

      1

      Specify the value from you obtained in the previous step for the .status.defaultChannel parameter.

      2

      You must specify the redhat-operators value.

   5. Create the Subscription object by running the following command:

      $ oc create -f pao-sub.yaml

   6. Change to the openshift-performance-addon-operator project:

      $ oc project openshift-performance-addon-operator

Procedure

1. Install the Performance Addon Operator using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose Performance Addon Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, under A specific namespace on the cluster select openshift-performance-addon-operator. Then, click Install.

2. Optional: Verify that the performance-addon-operator installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that Performance Addon Operator is listed in the openshift-operators project with a Status of Succeeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with a Succeeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, you can troubleshoot further:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the openshift-operators project.

Procedure

1. Run the following command:

   $ oc get csv

2. Review the output, checking the PHASE field. For example:

   VERSION    REPLACES                                         PHASE
   4.6.0      performance-addon-operator.v4.5.0                Installing
   4.5.0                                                       Replacing

3. Run get csv again to verify the output:

   # oc get csv

   Example output

   NAME                                DISPLAY                      VERSION   REPLACES                            PHASE
   performance-addon-operator.v4.5.0   Performance Addon Operator   4.6.0     performance-addon-operator.v4.5.0   Succeeded

1. Install Performance Addon Operator to the cluster.
2. Optional: Add a node to the OpenShift Container Platform cluster. See Setting BIOS parameters.
3. Add the label worker-rt to the worker nodes that require the real-time capability by using the oc command.

4. Create a new machine config pool for real-time nodes:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: worker-rt
     labels:
       machineconfiguration.openshift.io/role: worker-rt
   spec:
     machineConfigSelector:
       matchExpressions:
         - {
              key: machineconfiguration.openshift.io/role,
              operator: In,
              values: [worker, worker-rt],
           }
     paused: false
     nodeSelector:
       matchLabels:
         node-role.kubernetes.io/worker-rt: ""

   Note that a machine config pool worker-rt is created for group of nodes that have the label worker-rt.

5. Add the node to the proper machine config pool by using node role labels.

   Note

   You must decide which nodes are configured with real-time workloads. You could configure all of the nodes in the cluster, or a subset of the nodes. The Performance Addon Operator that expects all of the nodes are part of a dedicated machine config pool. If you use all of the nodes, you must point the Performance Addon Operator to the worker node role label. If you use a subset, you must group the nodes into a new machine config pool.

6. Create the PerformanceProfile with the proper set of housekeeping cores and realTimeKernel: enabled: true.

7. You must set machineConfigPoolSelector in PerformanceProfile:

     apiVersion: performance.openshift.io/v2
     kind: PerformanceProfile
     metadata:
      name: example-performanceprofile
     spec:
     ...
       realTimeKernel:
         enabled: true
       nodeSelector:
          node-role.kubernetes.io/worker-rt: ""
       machineConfigPoolSelector:
          machineconfiguration.openshift.io/role: worker-rt

8. Verify that a matching machine config pool exists with a label:

   $ oc describe mcp/worker-rt

   Example output

   Name:         worker-rt
   Namespace:
   Labels:       machineconfiguration.openshift.io/role=worker-rt

9. OpenShift Container Platform will start configuring the nodes, which might involve multiple reboots. Wait for the nodes to settle. This can take a long time depending on the specific hardware you use, but 20 minutes per node is expected.
10. Verify everything is working as expected.

Procedure

1. Create a pod with a QoS class of Guaranteed.
2. Optional: Disable CPU load balancing for DPDK.
3. Assign a proper node selector.

1. Create the pod:

   $ oc  apply -f qos-pod.yaml --namespace=qos-example

2. View detailed information about the pod:

   $ oc get pod qos-demo --namespace=qos-example --output=yaml

   Example output

   spec:
     containers:
       ...
   status:
     qosClass: Guaranteed

   Note

   If a container specifies its own memory limit, but does not specify a memory request, OpenShift Container Platform automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own CPU limit, but does not specify a CPU request, OpenShift Container Platform automatically assigns a CPU request that matches the limit.

1. Performing the mirroring.
2. Instructing the tests to consume the images from a custom registry.

1. The associated machine config pool is in a degraded state:

   # oc get mcp

   Example output

   NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master       rendered-master-2ee57a93fa6c9181b546ca46e1571d2d       True      False      False      3              3                   3                     0                      2d21h
   worker       rendered-worker-d6b2bdc07d9f5a59a6b68950acf25e5f       True      False      False      2              2                   2                     0                      2d21h
   worker-cnf   rendered-worker-cnf-6c838641b8a08fff08dbd8b02fb63f7c   False     True       True       2              1                   1                     1                      2d20h

2. The describe section of the MCP shows the reason:

   # oc describe mcp worker-cnf

   Example output

     Message:               Node node-worker-cnf is reporting: "prepping update:
     machineconfig.machineconfiguration.openshift.io \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not
     found"
       Reason:                1 nodes are reporting degraded status on sync

3. The degraded state should also appear under the performance profile status field marked as degraded = true:

   # oc describe performanceprofiles performance

   Example output

   Message: Machine config pool worker-cnf Degraded Reason: 1 nodes are reporting degraded status on sync.
   Machine config pool worker-cnf Degraded Message: Node yquinn-q8s5v-w-b-z5lqn.c.openshift-gce-devel.internal is
   reporting: "prepping update: machineconfig.machineconfiguration.openshift.io
   \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not found".    Reason:  MCPDegraded
      Status:  True
      Type:    Degraded

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to the Performance Addon Operator:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 1
   
    --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.6 2

   1

   The default OpenShift Container Platform must-gather image.

   2

   The must-gather image for low latency tuning diagnostics.

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

    $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.

Procedure

1. Create a namespace for the N3000 Operator by completing the following actions:

   1. Define the vran-acceleration-operators namespace by creating a file named n3000-namespace.yaml file as shown in the following example:

      apiVersion: v1
      kind: Namespace
      metadata:
          name: vran-acceleration-operators
          labels:
              openshift.io/cluster-monitoring: "true"

   2. Create the namespace by running the following command:

      $ oc create -f n3000-namespace.yaml

2. Install the N3000 Operator in the namespace you created in the previous step:

   1. Create the following OperatorGroup CR and save the YAML in the n3000-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
          name: n3000-operators
          namespace: vran-acceleration-operators
      spec:
          targetNamespaces:
          - vran-acceleration-operators

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f n3000-operatorgroup.yaml

   3. Run the following command to get the channel value required for the next step.

      $ oc get packagemanifest n3000 -n openshift-marketplace -o jsonpath='{.status.defaultChannel}'

      Example output

      stable

   4. Create the following Subscription CR and save the YAML in the n3000-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: n3000-subscription
          namespace: vran-acceleration-operators
      spec:
          channel: "<channel>"        1
          name: n3000
          source: certified-operators 2
          sourceNamespace: openshift-marketplace

      1

      Specify the value for channel from the value obtained in the previous step for the .status.defaultChannel parameter.

      2

      You must specify the certified-operators value.

   5. Create the Subscription CR by running the following command:

      $ oc create -f n3000-sub.yaml

Procedure

1. Install the OpenNESS Operator for Intel FPGA PAC N3000 by using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose OpenNESS Operator for Intel FPGA PAC N3000 from the list of available Operators, and then click Install.
   3. On the Install Operator page, select All namespaces on the cluster. Then, click Install.

2. Optional: Verify that the N3000 Operator is installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that OpenNESS Operator for Intel FPGA PAC N3000 is listed in the vran-acceleration-operators project with a Status of InstallSucceeded.

      Note

      During installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the console does not indicate that the Operator is installed, perform the following troubleshooting steps:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the vran-acceleration-operators project.

Procedure

1. Change to the vran-acceleration-operators project:

   $ oc project vran-acceleration-operators

2. Verify that the pods are running:

   $ oc get pods

   Example output

   NAME                                        READY       STATUS          RESTARTS    AGE
   fpga-driver-daemonset-8xz4c                 1/1         Running         0           15d
   fpgainfo-exporter-vhvdq                     1/1         Running         1           15d
   N3000-controller-manager-b68475c76-gcc6v    2/2         Running         1           15d
   N3000-daemonset-5k55l                       1/1         Running         1           15d
   N3000-discovery-blmjl                       1/1         Running         1           15d
   N3000-discovery-lblh7                       1/1         Running         1           15d

   The following section provides information on the installed pods:

   • fpga-driver-daemonset provides and loads the required Open Programmable Accelerator Engine (OPAE) drivers
   • fpgainfo-exporter provides N3000 telemetry data for Prometheus
   • N3000-controller-manager applies N3000Node CRs to the cluster and manages all the operand containers
   • N3000-daemonset is the main worker application. It monitors the changes in each node’s CR and acts on the changes. The logic implemented into this Daemon takes care of updating the cards' FPGA user image and NIC firmware. It is also responsible for draining the nodes and taking them out of commission when required by the update.
   • N3000-discovery discovers N3000 Accelerator devices installed and labels worker nodes if devices are present

3. Get all the nodes containing the Intel FPGA PAC N3000 card:

   $ oc get n3000node

   Example output

   NAME                       FLASH
   node1                      NotRequested

4. Get information about the card on each node:

   $ oc get n3000node node1 -o yaml

   Example output

   status:
     conditions:
     - lastTransitionTime: "2020-12-15T17:09:26Z"
       message: Inventory up to date
       observedGeneration: 1
       reason: NotRequested
       status: "False"
       type: Flashed
     fortville:
     - N3000PCI: 0000:1b:00.0
       NICs:
       - MAC: 64:4c:36:11:1b:a8
         NVMVersion: 7.00 0x800052b0 0.0.0
         PCIAddr: 0000:1a:00.0
         name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
       - MAC: 64:4c:36:11:1b:a9
         NVMVersion: 7.00 0x800052b0 0.0.0
         PCIAddr: 0000:1a:00.1
         name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
       - MAC: 64:4c:36:11:1b:ac
         NVMVersion: 7.00 0x800052b0 0.0.0
         PCIAddr: 0000:1c:00.0
         name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
       - MAC: 64:4c:36:11:1b:ad
         NVMVersion: 7.00 0x800052b0 0.0.0
         PCIAddr: 0000:1c:00.1
         name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
     fpga:
     - PCIAddr: 0000:1b:00.0 1
       bitstreamId: "0x23000410010310" 2
       bitstreamVersion: 0.2.3
       deviceId: "0x0b30"

   1

   The PCIAddr field indicates the PCI address of the card.

   2

   The bitstreamId field indicates the bitstream that is currently stored in flash.

5. Save the current bitstreamId, PCIAddr, the name, and the deviceId without "0x" padding.

   $ oc get n3000node -o json

6. Update the user bitstream of the Intel FPGA PAC N3000 card:

   1. Define the N3000 cluster resource to program by creating a file named n3000-cluster.yaml as shown in the following example:

      apiVersion: fpga.intel.com/v1
      kind: N3000Cluster
      metadata:
          name: n3000 1
          namespace: vran-acceleration-operators
      spec:
          nodes:
            - nodeName: "node1" 2
              fpga:
                - userImageURL: "http://10.10.10.122:8000/pkg/20ww27.5-2x2x25G-5GLDPC-v1.6.1-3.0.0_unsigned.bin" 3
                  PCIAddr: "0000:1b:00.0" 4
                  checksum: "0b0a87b974d35ea16023ceb57f7d5d9c" 5

      1

      Specify the name. The name must be n3000.

      2

      Specify the node to program.

      3

      Specify the URL for the user bitstream. This bitstream file must be accessible on an HTTP or HTTPS server.

      4

      Specify the PCI address of the card to program.

      5

      Specify the MD5 checksum of the bitstream that is specified in the userImageURL field.

      The N3000 daemon updates the FPGA user bitstream using the Open Programmable Acceleration Engine (OPAE) tools and resets the PCI device. The update of the FPGA user bitstream can require up to 40 minutes per card. For programming cards on multiple nodes, the programming happens one node at a time.

   2. Apply the update to begin programming the card with the bitstream:

      $ oc apply -f n3000-cluster.yaml

      The N3000 daemon starts programming the bitstream after the appropriate 5G FEC user bitstream has been provisioned, such as 20ww27.5-2x2x25G-5GLDPC-v1.6.1-3.0.0_unsigned.bin in this example, and after the CR has been created.

   3. Check the status:

      oc get n3000node

      Example output

      NAME             FLASH
      node1            InProgress

7. Check the logs:

   1. Determine the pod name of the N3000 daemon:

      $ oc get pod -o wide | grep n3000-daemonset | grep node1

      Example output

      n3000-daemonset-5k55l              1/1     Running   0          15d

   2. View the logs:

      $ oc logs n3000-daemonset-5k55l

      Example output

      ...
      {"level":"info","ts":1608054338.8866854,"logger":"daemon.drainhelper.cordonAndDrain()","msg":"node drained"}
      {"level":"info","ts":1608054338.8867319,"logger":"daemon.drainhelper.Run()","msg":"worker function - start"}
      {"level":"info","ts":1608054338.9003832,"logger":"daemon.fpgaManager.ProgramFPGAs","msg":"Start program","PCIAddr":"0000:1b:00.0"}
      {"level":"info","ts":1608054338.9004142,"logger":"daemon.fpgaManager.ProgramFPGA","msg":"Starting","pci":"0000:1b:00.0"}
      {"level":"info","ts":1608056309.9367146,"logger":"daemon.fpgaManager.ProgramFPGA","msg":"Program FPGA completed, start new power cycle N3000 ...","pci":"0000:1b:00.0"}
      {"level":"info","ts":1608056333.3528838,"logger":"daemon.drainhelper.Run()","msg":"worker function - end","performUncordon":true}
      ...

      The log file indicates the following flow of events:

      • The bitstream is downloaded and validated.
      • The node is drained and no workload is able to run during this time.

      • Flashing is started:

        • The bitstream is flashed into the card.
        • The bitstream is applied.
      • After flashing is complete the PCI device or devices on the node or nodes are reloaded. The OpenNESS SR-IOV Operator for Wireless FEC Accelerators is now able to find the new flashed device or devices.

Verification

1. Verify the status after the FPGA user bitstream update is complete:

   oc get n3000node

   Example output

   NAME             FLASH
   node1            Succeeded

2. Verify that the bitstream ID of the card has changed:

   oc get n3000node node1 -o yaml

   Example output

   status:
         conditions:
             - lastTransitionTime: "2020-12-15T18:18:53Z"
               message: Flashed successfully 1
               observedGeneration: 2
               reason: Succeeded
               status: "True"
               type: Flashed
         fortville:
         - N3000PCI: 0000:1b:00.0
           NICs:
           - MAC: 64:4c:36:11:1b:a8
             NVMVersion: 7.00 0x800052b0 0.0.0
             PCIAddr: 0000:1a:00.0
             name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
           - MAC: 64:4c:36:11:1b:a9
             NVMVersion: 7.00 0x800052b0 0.0.0
             PCIAddr: 0000:1a:00.1
             name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
           - MAC: 64:4c:36:11:1b:ac
             NVMVersion: 7.00 0x800052b0 0.0.0
             PCIAddr: 0000:1c:00.0
             name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
           - MAC: 64:4c:36:11:1b:ad
             NVMVersion: 7.00 0x800052b0 0.0.0
             PCIAddr: 0000:1c:00.1
             name: Ethernet Controller XXV710 Intel(R) FPGA Programmable Acceleration Card N3000 for Networking
         fpga:
         - PCIAddr: 0000:1b:00.0 2
           bitstreamId: "0x2315842A010601" 3
           bitstreamVersion: 0.2.3
           deviceId: "0x0b30" 4

   1

   The message field indicates the device is successfully flashed.

   2

   The PCIAddr field indicates the PCI address of the card.

   3

   The bitstreamId field indicates the updated bitstream ID.

   4

   The deviceID field indicates that device ID of the bitstream inside the card exposed to the system.

3. Check the FEC PCI devices on the node:

   1. Verify the node configuration is applied correctly:

      $ oc debug node/node1

      Expected output

      Starting pod/<node-name>-debug ...
      To use host binaries, run `chroot /host`
      
      Pod IP: <ip-address>
      If you don't see a command prompt, try pressing enter.
      
      sh-4.4#

   2. Verify that you can use the node file system:

      sh-4.4# chroot /host

      Expected output

      sh-4.4#

   3. List the PCI devices associated with the accelerator on your system:

      $ lspci | grep accelerators

      Expected output

      1b:00.0 Processing accelerators: Intel Corporation Device 0b30
      1d:00.0 Processing accelerators: Intel Corporation Device 0d8f (rev 01)

      Devices belonging to the FPGA are reported in the output. Device ID 0b30 is the RSU interface used to program the card, and the 0d8f is a physical function of the newly programmed 5G device.

Procedure

1. Create a namespace for the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by completing the following actions:

   1. Define the vran-acceleration-operators namespace by creating a file named sriov-namespace.yaml as shown in the following example:

      apiVersion: v1
      kind: Namespace
      metadata:
          name: vran-acceleration-operators
          labels:
             openshift.io/cluster-monitoring: "true"

   2. Create the namespace by running the following command:

      $ oc create -f sriov-namespace.yaml

2. Install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators in the namespace you created in the previous step by creating the following objects:

   1. Create the following OperatorGroup CR and save the YAML in the sriov-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
          name: vran-operators
          namespace: vran-acceleration-operators
      spec:
          targetNamespaces:
            - vran-acceleration-operators

   2. Create the OperatorGroup CR by running the following command:

      $ oc create -f sriov-operatorgroup.yaml

   3. Run the following command to get the channel value required for the next step.

      $ oc get packagemanifest sriov-fec -n openshift-marketplace -o jsonpath='{.status.defaultChannel}'

      Example output

      stable

   4. Create the following Subscription CR and save the YAML in the sriov-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
          name: sriov-fec-subscription
          namespace: vran-acceleration-operators
      spec:
          channel: "<channel>" 1
          name: sriov-fec
          source: certified-operators 2
          sourceNamespace: openshift-marketplace

      1

      Specify the value for channel from the value obtained in the previous step for the .status.defaultChannel parameter.

      2

      You must specify the certified-operators value.

   5. Create the Subscription CR by running the following command:

      $ oc create -f sriov-sub.yaml

Procedure

1. Install the OpenNESS SR-IOV Operator for Wireless FEC Accelerators by using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose OpenNESS SR-IOV Operator for Wireless FEC Accelerators from the list of available Operators, and then click Install.
   3. On the Install Operator page, select All namespaces on the cluster. Then, click Install.

2. Optional: Verify that the SRIOV-FEC Operator is installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that OpenNESS SR-IOV Operator for Wireless FEC Accelerators is listed in the vran-acceleration-operators project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the console does not indicate that the Operator is installed, perform the following troubleshooting steps:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the vran-acceleration-operators project.

Procedure

1. Change to the vran-acceleration-operators project:

   $ oc project vran-acceleration-operators

2. Verify that the SR-IOV-FEC Operator is installed:

   $ oc get csv -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                                        Phase
   sriov-fec.v1.1.0                            Succeeded
   n3000.v1.1.0                                Succeeded

3. Verify that the N3000 and sriov-fec pods are running:

   $  oc get pods

   Example output

   NAME                                            READY       STATUS      RESTARTS    AGE
   fpga-driver-daemonset-8xz4c                     1/1         Running     0           15d
   fpgainfo-exporter-vhvdq                         1/1         Running     1           15d
   N3000-controller-manager-b68475c76-gcc6v        2/2         Running     1           15d
   N3000-daemonset-5k55l                           1/1         Running     1           15d
   N3000-discovery-blmjl                           1/1         Running     1           15d
   N3000-discovery-lblh7                           1/1         Running     1           15d
   sriov-device-plugin-j5jlv                       1/1         Running     1           15d
   sriov-fec-controller-manager-85b6b8f4d4-gd2qg   1/1         Running     1           15d
   sriov-fec-daemonset-kqqs6                       1/1         Running     1           15d

   The following section provides information on the installed pods:

   • fpga-driver-daemonset provides and loads the required Open Programmable Accelerator Engine (OPAE) drivers
   • fpgainfo-exporter provides N3000 telemetry data for Prometheus
   • N3000-controller-manager applies N3000Node CRs to the cluster and manages all the operand containers
   • N3000-daemonset is the main worker application
   • N3000-discovery discovers N3000 Accelerator devices installed and labels worker nodes if devices are present
   • sriov-device-plugin expose the FEC virtual functions as resources under the node
   • sriov-fec-controller-manager applies CR to the node and maintains the operands containers

   • sriov-fec-daemonset is responsible for:

     • Discovering the SRIOV NICs on each node.
     • Syncing the status of the custom resource (CR) defined in step 6.
     • Taking the spec of the CR as input and configuring the discovered NICs.

4. Retrieve all the nodes containing one of the supported vRAN FEC accelerator devices:

   $ oc get sriovfecnodeconfig

   Example output

   NAME             CONFIGURED
   node1            Succeeded

5. Find the physical function (PF) of the SR-IOV FEC accelerator device to configure:

   $ oc get sriovfecnodeconfig node1 -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2021-03-19T17:19:37Z"
         message: Configured successfully
         observedGeneration: 1
         reason: ConfigurationSucceeded
         status: "True"
         type: Configured
       inventory:
           sriovAccelerators:
           - deviceID: 0d5c
             driver: ""
             maxVirtualFunctions: 16
             pciAddress: 0000.1d.00.0 1
             vendorID: "8086"
             virtualFunctions: [] 2

   1

   This field indicates the PCI Address of the card.

   2

   This field shows that the virtual functions are empty.

6. Configure the FEC device with the desired setting.

   1. Create the following custom resource (CR) and save the YAML in the sriovfec_n3000_cr.yaml file:

      apiVersion: sriovfec.intel.com/v1
      kind: SriovFecClusterConfig
      metadata:
          name: config
          namespace: vran-acceleration-operators
      spec:
        nodes:
          - nodeName: node1 1
            physicalFunctions:
              - pciAddress: 0000:1d:00.0 2
                pfDriver: pci-pf-stub
                vfDriver: vfio-pci
                vfAmount: 2 3
                bbDevConfig:
                  n3000:
                    # Network Type: either "FPGA_5GNR" or "FPGA_LTE"
                    networkType: "FPGA_5GNR"
                    pfMode: false
                    flrTimeout: 610
                    downlink:
                      bandwidth: 3
                      loadBalance: 128
                      queues: 4
                        vf0: 16
                        vf1: 16
                        vf2: 0
                        vf3: 0
                        vf4: 0
                        vf5: 0
                        vf6: 0
                        vf7: 0
                    uplink:
                      bandwidth: 3
                      loadBalance: 128
                      queues: 5
                        vf0: 16
                        vf1: 16
                        vf2: 0
                        vf3: 0
                        vf4: 0
                        vf5: 0
                        vf6: 0
                        vf7: 0

      1

      Specify the node name.

      2

      Specify the PCI Address of the card on which the SR-IOV-FEC Operator will be installed.

      3

      Specify the number of virtual functions. Create two virtual functions.

      4

      On vf0 create one queue with 16 buses (downlink and uplink).

      5

      On vf1 create one queue with 16 buses (downlink and uplink).

      Note

      For Intel PAC N3000 for vRAN Acceleration the user can create up to 8 VF devices. Each FEC PF device provides a total of 64 queues to be configured, 32 queues for uplink and 32 queues for downlink. The queues would be typically distributed evenly across the VFs.

   2. Apply the CR:

      $ oc apply -f sriovfec_n3000_cr.yaml

      After applying the CR, the SR-IOV FEC daemon starts configuring the FEC device.

Verification

1. Check the status:

   $ oc get sriovfecclusterconfig config -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2020-12-15T17:19:37Z"
         message: Configured successfully
         observedGeneration: 1
         reason: ConfigurationSucceeded
         status: "True"
         type: Configured
       inventory:
         sriovAccelerators:
         - deviceID: 0d8f
           driver: pci-pf-stub
           maxVirtualFunctions: 8
           pciAddress: 0000:1d:00.0
           vendorID: "8086"
           virtualFunctions:
           - deviceID: 0d90
             driver: vfio-pci
             pciAddress: 0000:1d:00.1
           - deviceID: 0d90
             driver: vfio-pci
             pciAddress: 0000:1d:00.2

2. Check the logs:

   1. Determine the name of the SR-IOV daemon pod:

      $ oc get pod | grep sriov-fec-daemonset

      Example output

      sriov-fec-daemonset-kqqs6                      1/1     Running   0          19h

   2. View the logs:

      $ oc logs sriov-fec-daemonset-kqqs6

      Example output

      2020-12-16T12:46:47.720Z        INFO    daemon.NodeConfigurator.applyConfig     configuring PF  {"requestedConfig": {"pciAddress":"0000:1d:00.0","pfDriver":"pci-pf-stub","vfDriver":"vfio-pci","vfAmount":2,"bbDevConfig":{"n3000":{
      "networkType":"FPGA_5GNR","pfMode":false,"flrTimeout":610,"downlink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}},"uplink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}}}}}}
      2020-12-16T12:46:47.720Z        INFO    daemon.NodeConfigurator.loadModule      executing command       {"cmd": "/usr/sbin/chroot /host/ modprobe pci-pf-stub"}
      2020-12-16T12:46:47.724Z        INFO    daemon.NodeConfigurator.loadModule      commands output {"output": ""}
      2020-12-16T12:46:47.724Z        INFO    daemon.NodeConfigurator.loadModule      executing command       {"cmd": "/usr/sbin/chroot /host/ modprobe vfio-pci"}
      2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator.loadModule      commands output {"output": ""}
      2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.0/driver_override"}
      2020-12-16T12:46:47.727Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/pci-pf-stub/bind"}
      2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.1/driver_override"}
      2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/vfio-pci/bind"}
      2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator device's driver_override path   {"path": "/sys/bus/pci/devices/0000:1d:00.2/driver_override"}
      2020-12-16T12:46:47.998Z        INFO    daemon.NodeConfigurator driver bind path        {"path": "/sys/bus/pci/drivers/vfio-pci/bind"}
      2020-12-16T12:46:47.999Z        INFO    daemon.NodeConfigurator.applyConfig     executing command       {"cmd": "/sriov_workdir/pf_bb_config FPGA_5GNR -c /sriov_artifacts/0000:1d:00.0.ini -p 0000:1d:00.0"}
      2020-12-16T12:46:48.017Z        INFO    daemon.NodeConfigurator.applyConfig     commands output {"output": "ERROR: Section (FLR) or name (flr_time_out) is not valid.
      FEC FPGA RTL v3.0
      UL.DL Weights = 3.3
      UL.DL Load Balance = 1
      28.128
      Queue-PF/VF Mapping Table = READY
      Ring Descriptor Size = 256 bytes
      
      --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
              |  PF | VF0 | VF1 | VF2 | VF3 | VF4 | VF5 | VF6 | VF7 |
      --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
      UL-Q'00 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'01 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'02 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'03 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'04 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'05 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'06 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'07 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'08 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'09 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'10 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'11 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'12 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'13 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'14 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'15 |     |  X  |     |     |     |     |     |     |     |
      UL-Q'16 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'17 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'18 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'19 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'20 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'21 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'22 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'23 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'24 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'25 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'26 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'27 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'28 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'29 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'30 |     |     |  X  |     |     |     |     |     |     |
      UL-Q'31 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'32 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'33 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'34 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'35 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'36 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'37 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'38 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'39 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'40 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'41 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'42 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'43 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'44 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'45 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'46 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'47 |     |  X  |     |     |     |     |     |     |     |
      DL-Q'48 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'49 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'50 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'51 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'52 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'53 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'54 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'55 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'56 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'57 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'58 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'59 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'60 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'61 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'62 |     |     |  X  |     |     |     |     |     |     |
      DL-Q'63 |     |     |  X  |     |     |     |     |     |     |
      --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
      
      Mode of operation = VF-mode
      FPGA_5GNR PF [0000:1d:00.0] configuration complete!"}
      2020-12-16T12:46:48.017Z        INFO    daemon.NodeConfigurator.enableMasterBus executing command       {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND"}
      2020-12-16T12:46:48.037Z        INFO    daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 = 0102\n"}
      2020-12-16T12:46:48.037Z        INFO    daemon.NodeConfigurator.enableMasterBus executing command       {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND=0106"}
      2020-12-16T12:46:48.054Z        INFO    daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 0106\n"}
      2020-12-16T12:46:48.054Z        INFO    daemon.NodeConfigurator.enableMasterBus MasterBus set   {"pci": "0000:1d:00.0", "output": "0000:1d:00.0 @04 0106\n"}
      2020-12-16T12:46:48.160Z        INFO    daemon.drainhelper.Run()        worker function - end   {"performUncordon": true}

3. Check the FEC configuration of the card:

   $ oc get sriovfecnodeconfig node1 -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2020-12-15T17:19:37Z"
         message: Configured successfully
         observedGeneration: 1
         reason: ConfigurationSucceeded
         status: "True"
         type: Configured
       inventory:
         sriovAccelerators:
         - deviceID: 0d8f 1
           driver: pci-pf-stub
           maxVirtualFunctions: 8
           pciAddress: 0000:1d:00.0
           vendorID: "8086"
         virtualFunctions:
         - deviceID: 0d90 2
           driver: vfio-pci
           pciAddress: 0000:1d:00.1
         - deviceID: 0d90
           driver: vfio-pci
           pciAddress: 0000:1d:00.2

   1

   0d8f is the deviceID physical function of the FEC device.

   2

   0d90 is the deviceID virtual function of the FEC device.

Procedure

1. Change to the vran-acceleration-operators project:

   $ oc project vran-acceleration-operators

2. Verify that the SR-IOV-FEC Operator is installed:

   $ oc get csv -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                                        Phase
   sriov-fec.v1.1.0                            Succeeded

3. Verify that the sriov-fec pods are running:

   $  oc get pods

   Example output

   NAME                                            READY       STATUS      RESTARTS    AGE
   sriov-device-plugin-j5jlv                       1/1         Running     1           15d
   sriov-fec-controller-manager-85b6b8f4d4-gd2qg   1/1         Running     1           15d
   sriov-fec-daemonset-kqqs6                       1/1         Running     1           15d

   • sriov-device-plugin expose the FEC virtual functions as resources under the node
   • sriov-fec-controller-manager applies CR to the node and maintains the operands containers

   • sriov-fec-daemonset is responsible for:

     • Discovering the SRIOV NICs on each node.
     • Syncing the status of the custom resource (CR) defined in step 6.
     • Taking the spec of the CR as input and configuring the discovered NICs.

4. Retrieve all the nodes containing one of the supported vRAN FEC accelerator devices:

   $ oc get sriovfecnodeconfig

   Example output

   NAME             CONFIGURED
   node1            Succeeded

5. Find the physical function (PF) of the SR-IOV FEC accelerator device to configure:

   $ oc get sriovfecnodeconfig node1 -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2021-03-19T17:19:37Z"
         message: Configured successfully
         observedGeneration: 1
         reason: ConfigurationSucceeded
         status: "True"
         type: Configured
       inventory:
          sriovAccelerators:
          - deviceID: 0d5c
            driver: ""
            maxVirtualFunctions: 16
            pciAddress: 0000:af:00.0 1
            vendorID: "8086"
            virtualFunctions: [] 2

   1

   This field indicates the PCI address of the card.

   2

   This field shows that the virtual functions are empty.

6. Configure the number of virtual functions and queue groups on the FEC device:

   1. Create the following custom resource (CR) and save the YAML in the sriovfec_acc100cr.yaml file:

      Note

      This example configures the ACC100 8/8 queue groups for 5G, 4 queue groups for Uplink, and another 4 queue groups for Downlink.

      apiVersion: sriovfec.intel.com/v1
      kind: SriovFecClusterConfig
      metadata:
        name: config 1
      spec:
        nodes:
         - nodeName: node1 2
           physicalFunctions:
             - pciAddress: 0000:af:00.0 3
               pfDriver: "pci-pf-stub"
               vfDriver: "vfio-pci"
               vfAmount: 16 4
               bbDevConfig:
                 acc100:
                   # Programming mode: 0 = VF Programming, 1 = PF Programming
                   pfMode: false
                   numVfBundles: 16
                   maxQueueSize: 1024
                   uplink4G:
                     numQueueGroups: 0
                     numAqsPerGroups: 16
                     aqDepthLog2: 4
                   downlink4G:
                    numQueueGroups: 0
                    numAqsPerGroups: 16
                    aqDepthLog2: 4
                   uplink5G:
                    numQueueGroups: 4
                    numAqsPerGroups: 16
                    aqDepthLog2: 4
                   downlink5G:
                    numQueueGroups: 4
                    numAqsPerGroups: 16
                    aqDepthLog2: 4

      1

      Specify a name for the CR object. The only name that can be specified is config.

      2

      Specify the node name.

      3

      Specify the PCI address of the card on which the SR-IOV-FEC Operator will be installed.

      4

      Specify the number of virtual functions to create. For the Intel vRAN Dedicated Accelerator ACC100, create all 16 VFs.

      Note

      The card is configured to provide up to 8 queue groups with up to 16 queues per group. The queue groups can be divided between groups allocated to 5G and 4G and Uplink and Downlink. The Intel vRAN Dedicated Accelerator ACC100 can be configured for:

      • 4G or 5G only
      • 4G and 5G at the same time

      Each configured VF has access to all the queues. Each of the queue groups have a distinct priority level. The request for a given queue group is made from the application level that is, the vRAN application leveraging the FEC device.

   2. Apply the CR:

      $ oc apply -f sriovfec_acc100cr.yaml

      After applying the CR, the SR-IOV FEC daemon starts configuring the FEC device.

Verification

1. Check the status:

   $ oc get sriovfecclusterconfig config -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2021-03-19T11:46:22Z"
         message: Configured successfully
         observedGeneration: 1
         reason: Succeeded
         status: "True"
         type: Configured
       inventory:
         sriovAccelerators:
         - deviceID: 0d5c
           driver: pci-pf-stub
           maxVirtualFunctions: 16
           pciAddress: 0000:af:00.0
           vendorID: "8086"
           virtualFunctions:
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.0
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.1
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.2
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.3
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.4

2. Check the logs:

   1. Determine the pod name of the SR-IOV daemon:

      $ oc get po -o wide | grep sriov-fec-daemonset | grep node1

      Example output

      sriov-fec-daemonset-kqqs6                      1/1     Running   0          19h

   2. View the logs:

      $ oc logs sriov-fec-daemonset-kqqs6

      Example output

      {"level":"Level(-2)","ts":1616794345.4786215,"logger":"daemon.drainhelper.cordonAndDrain()","msg":"node drained"}
      {"level":"Level(-4)","ts":1616794345.4786265,"logger":"daemon.drainhelper.Run()","msg":"worker function - start"}
      {"level":"Level(-4)","ts":1616794345.5762916,"logger":"daemon.NodeConfigurator.applyConfig","msg":"current node status","inventory":{"sriovAccelerat
      ors":[{"vendorID":"8086","deviceID":"0b32","pciAddress":"0000:20:00.0","driver":"","maxVirtualFunctions":1,"virtualFunctions":[]},{"vendorID":"8086"
      ,"deviceID":"0d5c","pciAddress":"0000:af:00.0","driver":"","maxVirtualFunctions":16,"virtualFunctions":[]}]}}
      {"level":"Level(-4)","ts":1616794345.5763638,"logger":"daemon.NodeConfigurator.applyConfig","msg":"configuring PF","requestedConfig":{"pciAddress":"
      0000:af:00.0","pfDriver":"pci-pf-stub","vfDriver":"vfio-pci","vfAmount":2,"bbDevConfig":{"acc100":{"pfMode":false,"numVfBundles":16,"maxQueueSize":1
      024,"uplink4G":{"numQueueGroups":4,"numAqsPerGroups":16,"aqDepthLog2":4},"downlink4G":{"numQueueGroups":4,"numAqsPerGroups":16,"aqDepthLog2":4},"uplink5G":{"numQueueGroups":0,"numAqsPerGroups":16,"aqDepthLog2":4},"downlink5G":{"numQueueGroups":0,"numAqsPerGroups":16,"aqDepthLog2":4}}}}}
      {"level":"Level(-4)","ts":1616794345.5774765,"logger":"daemon.NodeConfigurator.loadModule","msg":"executing command","cmd":"/usr/sbin/chroot /host/ modprobe pci-pf-stub"}
      {"level":"Level(-4)","ts":1616794345.5842702,"logger":"daemon.NodeConfigurator.loadModule","msg":"commands output","output":""}
      {"level":"Level(-4)","ts":1616794345.5843055,"logger":"daemon.NodeConfigurator.loadModule","msg":"executing command","cmd":"/usr/sbin/chroot /host/ modprobe vfio-pci"}
      {"level":"Level(-4)","ts":1616794345.6090655,"logger":"daemon.NodeConfigurator.loadModule","msg":"commands output","output":""}
      {"level":"Level(-2)","ts":1616794345.6091156,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:af:00.0/driver_override"}
      {"level":"Level(-2)","ts":1616794345.6091807,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/pci-pf-stub/bind"}
      {"level":"Level(-2)","ts":1616794345.7488534,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:b0:00.0/driver_override"}
      {"level":"Level(-2)","ts":1616794345.748938,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/vfio-pci/bind"}
      {"level":"Level(-2)","ts":1616794345.7492096,"logger":"daemon.NodeConfigurator","msg":"device's driver_override path","path":"/sys/bus/pci/devices/0000:b0:00.1/driver_override"}
      {"level":"Level(-2)","ts":1616794345.7492566,"logger":"daemon.NodeConfigurator","msg":"driver bind path","path":"/sys/bus/pci/drivers/vfio-pci/bind"}
      {"level":"Level(-4)","ts":1616794345.74968,"logger":"daemon.NodeConfigurator.applyConfig","msg":"executing command","cmd":"/sriov_workdir/pf_bb_config ACC100 -c /sriov_artifacts/0000:af:00.0.ini -p 0000:af:00.0"}
      {"level":"Level(-4)","ts":1616794346.5203931,"logger":"daemon.NodeConfigurator.applyConfig","msg":"commands output","output":"Queue Groups: 0 5GUL, 0 5GDL, 4 4GUL, 4 4GDL\nNumber of 5GUL engines 8\nConfiguration in VF mode\nPF ACC100 configuration complete\nACC100 PF [0000:af:00.0] configuration complete!\n\n"}
      {"level":"Level(-4)","ts":1616794346.520459,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"executing command","cmd":"/usr/sbin/chroot /host/ setpci -v -s 0000:af:00.0 COMMAND"}
      {"level":"Level(-4)","ts":1616794346.5458736,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"commands output","output":"0000:af:00.0 @04 = 0142\n"}
      {"level":"Level(-4)","ts":1616794346.5459251,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"executing command","cmd":"/usr/sbin/chroot /host/ setpci -v -s 0000:af:00.0 COMMAND=0146"}
      {"level":"Level(-4)","ts":1616794346.5795262,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"commands output","output":"0000:af:00.0 @04 0146\n"}
      {"level":"Level(-2)","ts":1616794346.5795407,"logger":"daemon.NodeConfigurator.enableMasterBus","msg":"MasterBus set","pci":"0000:af:00.0","output":"0000:af:00.0 @04 0146\n"}
      {"level":"Level(-4)","ts":1616794346.6867144,"logger":"daemon.drainhelper.Run()","msg":"worker function - end","performUncordon":true}
      {"level":"Level(-4)","ts":1616794346.6867719,"logger":"daemon.drainhelper.Run()","msg":"uncordoning node"}
      {"level":"Level(-4)","ts":1616794346.6896322,"logger":"daemon.drainhelper.uncordon()","msg":"starting uncordon attempts"}
      {"level":"Level(-2)","ts":1616794346.69735,"logger":"daemon.drainhelper.uncordon()","msg":"node uncordoned"}
      {"level":"Level(-4)","ts":1616794346.6973662,"logger":"daemon.drainhelper.Run()","msg":"cancelling the context to finish the leadership"}
      {"level":"Level(-4)","ts":1616794346.7029872,"logger":"daemon.drainhelper.Run()","msg":"stopped leading"}
      {"level":"Level(-4)","ts":1616794346.7030034,"logger":"daemon.drainhelper","msg":"releasing the lock (bug mitigation)"}
      {"level":"Level(-4)","ts":1616794346.8040674,"logger":"daemon.updateInventory","msg":"obtained inventory","inv":{"sriovAccelerators":[{"vendorID":"8086","deviceID":"0b32","pciAddress":"0000:20:00.0","driver":"","maxVirtualFunctions":1,"virtualFunctions":[]},{"vendorID":"8086","deviceID":"0d5c","pciAddress":"0000:af:00.0","driver":"pci-pf-stub","maxVirtualFunctions":16,"virtualFunctions":[{"pciAddress":"0000:b0:00.0","driver":"vfio-pci","deviceID":"0d5d"},{"pciAddress":"0000:b0:00.1","driver":"vfio-pci","deviceID":"0d5d"}]}]}}
      {"level":"Level(-4)","ts":1616794346.9058325,"logger":"daemon","msg":"Update ignored, generation unchanged"}
      {"level":"Level(-2)","ts":1616794346.9065044,"logger":"daemon.Reconcile","msg":"Reconciled","namespace":"vran-acceleration-operators","name":"pg-itengdvs02r.altera.com"}

3. Check the FEC configuration of the card:

   $ oc get sriovfecnodeconfig node1 -o yaml

   Example output

   status:
       conditions:
       - lastTransitionTime: "2021-03-19T11:46:22Z"
         message: Configured successfully
         observedGeneration: 1
         reason: Succeeded
         status: "True"
         type: Configured
       inventory:
         sriovAccelerators:
         - deviceID: 0d5c 1
           driver: pci-pf-stub
           maxVirtualFunctions: 16
           pciAddress: 0000:af:00.0
           vendorID: "8086"
           virtualFunctions:
           - deviceID: 0d5d 2
             driver: vfio-pci
             pciAddress: 0000:b0:00.0
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.1
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.2
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.3
           - deviceID: 0d5d
             driver: vfio-pci
             pciAddress: 0000:b0:00.4

   1

   The value 0d5c is the deviceID physical function of the FEC device.

   2

   The value 0d5d is the deviceID virtual function of the FEC device.

Procedure

1. Create a namespace for the test by completing the following actions:

   1. Define the test-bbdev namespace by creating a file named test-bbdev-namespace.yaml file as shown in the following example:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: test-bbdev
        labels:
          openshift.io/run-level: "1"

   2. Create the namespace by running the following command:

      $ oc create -f test-bbdev-namespace.yaml

2. Create the following Pod specification, and then save the YAML in the pod-test.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: pod-bbdev-sample-app
     namespace: test-bbdev 1
   spec:
     containers:
     - securityContext:
         privileged: false
         capabilities:
           add:
            - IPC_LOCK
            - SYS_NICE
       name: bbdev-sample-app
       image: bbdev-sample-app:1.0  2
       command: [ "sudo", "/bin/bash", "-c", "--" ]
       runAsUser: 0 3
       resources:
         requests:
           hugepages-1Gi: 4Gi 4
           memory: 1Gi
           cpu: "4" 5
           intel.com/intel_fec_5g: '1' 6
           #intel.com/intel_fec_acc100: '1'
           #intel.com/intel_fec_lte: '1'
         limits:
           memory: 4Gi
           cpu: "4"
           hugepages-1Gi: 4Gi
           intel.com/intel_fec_5g: '1'
           #intel.com/intel_fec_acc100: '1'
           #intel.com/intel_fec_lte: '1

   1

   Specify the namespace you created in step 1.

   2

   This defines the test image containing the compiled DPDK.

   3

   Make the container execute internally as the root user.

   4

   Specify hugepage size hugepages-1Gi and the quantity of hugepages that will be allocated to the pod. Hugepages and isolated CPUs need to be configured using the Performance Addon Operator.

   5

   Specify the number of CPUs.

   6

   Testing of the N3000 5G FEC configuration is supported by intel.com/intel_fec_5g.

   Note

   To test the ACC100 configuration, uncomment intel.com/intel_fec_acc100 by removing the # symbol. To test the N3000 4G/LTE configuration, uncomment intel.com/intel_fec_lte by removing the # symbol. Only one resource can be active at any time.

3. Create the pod:

   $ oc apply -f pod-test.yaml

4. Check that the pod is created:

   $ oc get pods -n test-bbdev

   Example output

   NAME                                            READY           STATUS          RESTARTS        AGE
   pod-bbdev-sample-app                            1/1             Running         0               80s

5. Use a remote shell to log in to the pod-bbdev-sample-app:

   $ oc rsh pod-bbdev-sample-app

   Example output

   sh-4.4#

6. Print a list of environment variables:

   sh-4.4# env

   Example output

   N3000_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR=172.30.133.131
   SRIOV_FEC_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp
   DPDK_VERSION=20.11
   PCIDEVICE_INTEL_COM_INTEL_FEC_5G=0.0.0.0:1d.00.0 1
   ~/usr/bin/env
   HOSTNAME=fec-pod

   1

   This is the PCI address of the virtual function. Depending on the resource that you requested in the pod-test.yaml file, this can be any one of following three PCI addresses:

   • PCIDEVICE_INTEL_COM_INTEL_FEC_ACC100
   • PCIDEVICE_INTEL_COM_INTEL_FEC_5G
   • PCIDEVICE_INTEL_COM_INTEL_FEC_LTE

7. Change to the test-bbdev directory:

   sh-4.4# cd test/test-bbdev/

   Note

   The directory is in the pod and not on your local computer.

8. Check the CPUs that are assigned to the pod:

   sh-4.4# export CPU=$(cat /sys/fs/cgroup/cpuset/cpuset.cpus)
   sh-4.4# echo ${CPU}

   This prints out the CPUs that are assigned to the fec.pod.

   Example output

   24,25,64,65

9. Run the test-bbdev application to test the device:

   sh-4.4# ./test-bbdev.py -e="-l ${CPU} -a ${PCIDEVICE_INTEL_COM_INTEL_FEC_5G}" -c validation \ -n 64 -b 32 -l 1 -v ./test_vectors/*"

   Example output

   Executing: ../../build/app/dpdk-test-bbdev -l 24-25,64-65 0000:1d.00.0 -- -n 64 -l 1 -c validation -v ./test_vectors/bbdev_null.data -b 32
   EAL: Detected 80 lcore(s)
   EAL: Detected 2 NUMA nodes
   Option -w, --pci-whitelist is deprecated, use -a, --allow option instead
   EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
   EAL: Selected IOVA mode 'VA'
   EAL: Probing VFIO support...
   EAL: VFIO support initialized
   EAL:   using IOMMU type 1 (Type 1)
   EAL: Probe PCI driver: intel_fpga_5ngr_fec_vf (8086:d90) device: 0000:1d.00.0 (socket 1)
   EAL: No legacy callbacks, legacy socket not created
   
   
   
   ===========================================================
   Starting Test Suite : BBdev Validation Tests
   Test vector file = ldpc_dec_v7813.data
   Device 0 queue 16 setup failed
   Allocated all queues (id=16) at prio0 on dev0
   Device 0 queue 32 setup failed
   Allocated all queues (id=32) at prio1 on dev0
   Device 0 queue 48 setup failed
   Allocated all queues (id=48) at prio2 on dev0
   Device 0 queue 64 setup failed
   Allocated all queues (id=64) at prio3 on dev0
   Device 0 queue 64 setup failed
   All queues on dev 0 allocated: 64
   + ------------------------------------------------------- +
   == test: validation
   dev:0000:b0:00.0, burst size: 1, num ops: 1, op type: RTE_BBDEV_OP_LDPC_DEC
   Operation latency:
           avg: 23092 cycles, 10.0838 us
           min: 23092 cycles, 10.0838 us
           max: 23092 cycles, 10.0838 us
   TestCase [ 0] : validation_tc passed
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +
    + Test Suite Summary : BBdev Validation Tests
    + Tests Total :        1
    + Tests Skipped :      0
    + Tests Passed :       1 1
    + Tests Failed :       0
    + Tests Lasted :       177.67 ms
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +

   1

   While some tests can be skipped, be sure that the vector tests pass.