[RHOCP 4] NorthboundStale Alert Triggered and Pods Failing to Spawn in the Cluster

Issue

• Getting NorthboundStale alert on OpenShift console as Critical.
• Getting below error when trying to deploy a pod onto the cluster.

Netns:"/var/run/netns/xxxxxx-d154-xxxx-b513-xxxxxx" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-logging;K8S_POD_NAME=elasticsearch-xxxxxx;K8S_POD_INFRA_CONTAINER_ID=609a26bc7109e558b538131dbd3738333333b7e3362d96d0ff65780e4a401bf9;K8S_POD_UID=xxxxxx-806a-4e02-xxxxxxx-xxxxxx" Path:"", result: "", err: error configuring pod [openshift-logging/elasticsearch-xxxxxx-xxxxxxx] networking: [openshift-logging/elasticsearch-xxxxxx/xxxxxxx:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-logging/elasticsearch-xxxxxx/xxxxxxx:ovn-kubernetes xxxxxxxxxxxx network default NAD default] [openshift-logging/elasticsearch-xxxxxx/xxxxxxx:ovn-kubernetes xxxxxxxxxxxx network default NAD default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded

• Multus PODs are keep complaining about CNI ADD requested failed, this because CNI add requests are never reaching to ovn-master PODs.

ovn-k8s-cni-overlay","ipam":{},"dns":{},"logFile":"/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log","logLevel":"4","logfile-maxsize":100,"logfile-maxbackups":5,"logfile-maxage":0}
2024-05-19T06:36:17.xxxxxxx 2024-05-19T06:36:17Z [error] [openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx xxxxxxxxxxxxxxxx network default NAD default] [openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx 609a26bc7109e558b538131dbd3738xxxxxxxxxxxxxxxxx network default NAD default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded
2024-05-19T06:36:17.824711617Z '
2024-05-19T06:36:17.824761540Z 2024-05-19T06:36:17Z [verbose] ADD finished CNI request ContainerID:"xxxxxxxx3738333333b7e3362d96d0ff657xxxxxxxxxx" Netns:"/var/run/netns/xxxxxxx-d154-xxxxx-b513-xxxxxxx" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-logging;K8S_POD_NAME=elasticsearch-xxxxxxx/xxxxxxxx;K8S_POD_INFRA_CONTAINER_ID=xxxxxxxxxxx;K8S_POD_UID=xxxxxx-806a-4e02-b69b-xxxxxx" Path:"", result: "", err: error configuring pod [openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx] networking: [openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx/xxxxxxx:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-logging/elasticsearch-im-app-28601475-t5ndd 609a26bc7109e558b538131dbd3738333333bxxxxxxxxxxx network default NAD default] [openshift-logging/elasticsearch-xxxxxxx/xxxxxxxx xxxxxxxx8b538131dbd37383xxxxxx7e3362d96d0ff657xxxxxxx network default NAD default] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded