Error in setting annotation on pod due to invalid spec.containers

Issue

• ovnkube-master pod logs the following events for the specific POD.

024-04-22T04:09:14.936484347+00:00 stderr F E0422 04:09:14.934684 1 kube.go:80] Error in setting annotation on pod testnamespace/example-pod-xxx: Pod "example-pod-xxx" is invalid: spec.containers: Forbidden: pod updates may not add or remove containers

• Kubelet throws the following events for specific pod in the pod's namespace.

Warning   ErrorAddingLogicalPort   pod/example-pod-xxx   addLogicalPort failed for testnamespace/example-pod-xxx: failed to set annotation on pod example-pod-xxx: Pod "example-pod-xxx" is invalid: spec.containers: Forbidden: pod updates may not add or remove containers

Warning   FailedCreatePodSandBox   pod/example-pod-xxx   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_example-pod-xxx_testnamespace_1d39c9ea-cc27-4d58-82d6-7536a64e31c0_0(xxxxxx): error adding pod testnamespace_example-pod-xxx to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [testnamespace/example-pod-xxx/xxxxxx:ovn-kubernetes]: error adding container to network "ovn-kubernetes": failed to send CNI request: Post "http://dummy/": dial unix /var/run/ovn-kubernetes/cni//ovn-cni-server.sock: connect: connection refused

Warning   FailedCreatePodSandBox   pod/example-pod-xxx  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_example-pod-xxx_testnamespace_1d39c9ea-cc27-4d58-82d6-7536a64e31c0_0(xxxxxxx): error adding pod testnamespace_example-pod-xxx to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [testnamespace/example-pod-xxx/1d39c9ea-cc27-4d58-82d6-7536a64e31c0:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[testnamespace/example-pod-xxx] [testnamespace/example-pod-xxx] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded...

• pod creation fails with Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox