[RHOCP 4] Pods are stuck in ContainerCreating state: http2: client connection lost

Issue

• When deployed pods get stuck in ContainerCreating state. Containers in the pod are stuck in waiting status and the pod never successfully gets an IP assigned.

• Events from the affected namespace.

  $ oc get events -n <affected-namespace> (openshift-storage in this example)
  
  35s        Warning  FailedCreatePodSandBox  pod/noobaa-operator-xxx      (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_noobaa-operator-xxx_openshift-storage_4658ed69-8dbb-49ef-b6fc-6575c7b59b43_0(b04981f5be63470994467292c077505bb8b9f78c5379737b5997bbaa415f80de): error adding pod openshift-storage_noobaa-operator-xxx to CNI network "multus-cni-network": Multus: [openshift-storage/noobaa-operator-xxx]: error setting the networks status: SetNetworkStatus: failed to update the pod noobaa-operator-xxx in out of cluster comm: SetNetworkStatus: failed to update the pod noobaa-operator-xxx in out of cluster comm: status update failed for pod openshift-storage/noobaa-operator-xxx: Put "https://[api-int.xxx]:6443/api/v1/namespaces/openshift-storage/pods/noobaa-operator-xxx/status?timeout=1m0s": http2: client connection lost
  

• If using OVN as the CNI. look at the OVNkube node logs on the affected node. Will repeatedly see this pattern.

  CNI ADD completes
  15:01:09.711165    4536 cni.go:248] [<namespace>/<pod-name>] ADD finished CNI request [<namespace>/<pod-name>], result "{\"interfaces\":[{\"name\":\" <netns_id>\",\"mac\":\"xx:xx:xx:xx:xx:xx\"},{\"name\":\"eth0\",\"mac\":\"xx:xx:xx:xx:xx:xx\",\"sandbox\":\"/var/run/netns/afbab413-4a03-4e82-8877-c61a719ef663\"}],\"ips\":[{\"interface\":1,\"address\":\"xx.xxx.xxx.x/23\",\"gateway\":\"xx.xxx.xxx.x\"}],\"dns\":{}}", err <nil>     
  
  45s later      
  CNI DEL is called:
  15:01:54.908021    4536 cni.go:227][<namespace>/<pod-name>] DEL finished CNI request [<namespace>/<pod-name>], result "{\"dns\":{}}", err <nil>
  
  CNI ADD invoked again                                                                         
  15:01:56.055706    4536 cni.go:227] [<namespace>/<pod-name>] ADD starting CNI request [<namespace>/<pod-name>]