apiserver pod in CrashLoopBackOff in OpenShift 4

Issue

• One of the apiserver pod is in CrashLoopBackOff:

  # oc get pods -n openshift-apiserver
  NAME              READY   STATUS             RESTARTS   AGE
  apiserver-6ttpq   1/1     Running            0          2d8h
  apiserver-9zx6c   1/1     Running            0          2d8h
  apiserver-kcztv   0/1     CrashLoopBackOff   717        2d7h
  

• Logs from the apiserver pod:

  I0429 10:54:27.211999       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd.openshift-etcd.svc:2379 0  <nil>}]
  I0429 10:54:37.235303       1 store.go:1342] Monitoring deploymentconfigs.apps.openshift.io count at <storage-prefix>//deploymentconfigs
  I0429 10:54:37.363091       1 client.go:361] parsed scheme: "endpoint"
  I0429 10:54:37.363131       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://etcd.openshift-etcd.svc:2379 0  <nil>}]
  F0429 10:54:57.363416       1 openshift_apiserver.go:420] error building REST storage: context deadline exceeded
  W0429 10:54:57.363447       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://etcd.openshift-etcd.svc:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
  

• Upgrade stuck and authentication cluster operator reports crashlooping:

  $ oc get clusteroperator
  NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
  authentication                             4.10.36   True        False         True       24s     APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (crashlooping container is waiting in apiserver-a0a0a0a0a0-00aaa pod)...