Multus and OVN pods are in unhealthy state in RHOCP 4

Issue

• Pod is stuck in a pending state due to multus and ovn pods were not healthy and the pods with the pending state show below warning logs and events:

  1h56m      Warning  FailedCreatePodSandBox        pod/apiserver-xxxxx   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-xxxxx_openshift-oauth-apiserver_1234xx-567xx-89xx-0xxxxx_0(784b754b671b637811edf1e9210b613d22b31c19746a7c137e1350dba8187159): error adding pod openshift-oauth-apiserver_apiserver-xxxxx to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [openshift-oauth-apiserver/apiserver-xxxxx/1234xx-567xx-89xx-0xxxxx:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-oauth-apiserver/apiserver-xxxxx 784b754b671b637811edf1e9210b613d22b31c19746a7c137e1350dba8187159] [openshift-oauth-apiserver/apiserver-xxxxx 784b754b671b637811edf1e9210b613d22b31c19746a7c137e1350dba8187159] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
  

• Multus-admission-controller and network-metrics-daemon pods is showing an unhealthy state with 0 out 2 containers running:

  multus-admission-controller-xxxx    0/2    Running    3         31d                  master3.example.com
  network-metrics-daemon-xxxx        0/2    Running    3         31d                   master3.example.com