Cluster Operator unavailable with 'APIServicesAvailable: "apps.openshift.io.v1" is not ready'

Solution Verified - Updated -

Issue

  • The OpenShift Container Platform Cluster Operator openshift-apiserver is not available and is showing the following APIServices_Error errors:

    APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
    
  • The OpenShift API Server (apiserver) Pod in the openshift-apiserver namespace is showing the following connection refused error messages:

    2023-06-19T07:59:22.838495962Z W0619 07:59:22.838437       1 logging.go:59] [core] [Channel #159 SubChannel #160] grpc: addrConn.createTransport failed to connect to {
    2023-06-19T07:59:22.838495962Z   "Addr": "10.140.20.23:2379",
    2023-06-19T07:59:22.838495962Z   "ServerName": "10.140.20.23",
    2023-06-19T07:59:22.838495962Z   "Attributes": null,
    2023-06-19T07:59:22.838495962Z   "BalancerAttributes": null, 
    2023-06-19T07:59:22.838495962Z   "Type": 0,
    2023-06-19T07:59:22.838495962Z   "Metadata": null
    2023-06-19T07:59:22.838495962Z }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.140.20.23:2379: connect: connection refused"
    
  • The etcd Pods show error messages similar to the following:

    2023-06-19T09:34:37.366653439Z {"level":"warn","ts":"2023-06-19T09:34:37.366Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"ed898fb22814c213","remote-peer-id-stream-handler":"fd898fb22814c212","remote-peer-id-from":"32b626d55668f0af"}
    2023-06-19T09:34:37.367468970Z {"level":"warn","ts":"2023-06-19T09:34:37.367Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"ed898fb22814c213","remote-peer-id-stream-handler":"fd898fb22814c212","remote-peer-id-from":"32b626d55668f0af"}
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.12

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content