IB-SRIOV CNI compatibility mismatch

Solution In Progress - Updated -

Environment

  • OpenShift Container Platform 4.16 or older with Using SR-IOV(IB) plugin

Issue

In OpenShift Container Platform 4.16 or older, It is unable to create Pods due to a CNI compatibility mismatch error.

Resolution

Created the patch in upstream for resolving this issue.
This issue is handling in the OCPBUGS-63662

This patch was already taken into the OpenShift Container Platform 4.17 or newer.
This issue is occurred only in OpenShift Container Platform 4.16 or older

Root Cause

It caused by CNI compatibility mismatch, where the NetworkAttachmentDefinition (NAD) generated by SR-IOV specifies cniVersion: 1.0.0, but the actual CNI plugins only support up to version 0.4.0.
The ib-sriov binary version on a worker node was found to be 1.0.0, which is not supported by the current plugins.

Workaround:
1. Check existing NAD created by sriovibNetworks.
2. Remove the sriovibNetworks.
3. Create the NAD manually with cniVersion under 0.4.0.

Diagnostic Steps

When creating the pod, the following message is seen in the event log.

Warning  FailedCreatePodSandBox  11s   kubelet    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox xx-xx-xx-xx-xx(xx): error adding pod xx to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): 
CNI request failed with status 400: 'ContainerID:"xx" Netns:"/var/run/netns/xx" IfName:"xx" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=xx;K8S_POD_NAME=xx;K8S_POD_INFRA_CONTAINER_ID=xxx;K8S_POD_UID=xxx" Path:"" ERRORED: error configuring pod [xx] networking: [xxx]: error adding container to network "xx-1": incompatible CNI versions; config is "1.0.0", plugin supports ["0.1.0" "0.2.0" "0.3.0" "0.3.1" "0.4.0"]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments