Pods are not able to reach the kubernetes service

Solution Verified - Updated -

Issue

Various pods are in CrashLoopBackOff status:

$ oc get pods -A -o wide | grep -v -e Running -e Completed
NAMESPACE                                          NAME                                                              READY   STATUS             RESTARTS   AGE     IP             NODE
openshift-authentication-operator                  authentication-operator-df9d6885b-gnlfb                           0/1     CrashLoopBackOff   87         7h3m    10.128.0.42    master-0
openshift-cluster-node-tuning-operator             cluster-node-tuning-operator-5fbf9968bd-jznqr                     0/1     CrashLoopBackOff   86         7h10m   10.128.0.43    master-0
openshift-console                                  console-74f997dcdb-bfqkh                                          0/1     CrashLoopBackOff   771        3d      10.128.0.25    master-0
openshift-kube-controller-manager-operator         kube-controller-manager-operator-559f8f95bb-ll6bn                 0/1     CrashLoopBackOff   78         7h6m    10.128.0.45    master-0
openshift-operator-lifecycle-manager               catalog-operator-5fb9f7c6c5-94s5r                                 0/1     CrashLoopBackOff   92         7h10m   10.128.0.40    master-0
openshift-operator-lifecycle-manager               package-server-manager-db768d594-b528r                            0/1     CrashLoopBackOff   86         7h10m   10.128.0.41    master-0

A closer look at their logs shows impossibility to reach the internal kubernetes service:

$ oc -n openshift-authentication-operator logs authentication-operator-df9d6885b-gnlfb
...
2022-11-21T18:32:05.729187953Z F1121 18:32:05.729135       1 cmd.go:138] unable to load configmap based request-header-client-ca-file: Get "https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 172.30.0.1:443: connect: no route to host
...

$ oc -n openshift-kube-controller-manager-operator logs kube-controller-manager-operator-559f8f95bb-ll6bn
...
2022-11-21T18:31:56.576209254Z F1121 18:31:56.576149       1 cmd.go:138] unable to load configmap based request-header-client-ca-file: Get "https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 172.30.0.1:443: connect: no route to host
...

Environment

  • Red Hat Openshift Container Platform
    • 4.11
  • Azure Red Hat OpenShift

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content