openshift-apiserver degraded and kube-apiserver not available in OCP 4

Solution In Progress - Updated -

Issue

  • The openshift-apiserver Cluster Operator is degraded or not available:

    $ oc get co openshift-apiserver
    NAME                 VERSION  AVAILABLE  PROGRESSING  DEGRADED  SINCE
    openshift-apiserver  4.5.24   False      False        False     1h11m
    
  • There are "panic" errors in the kube-apiserver pods:

    2021-01-01T00:00:00.000000000Z E0101 00:00:00.000000      18 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started)
    
  • The status of the openshift-apiserver Cluster Operator has the following message:

    - lastTransitionTime: '2021-01-29T02:27:11Z'
      message: 'APIServicesAvailable: "apps.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/apps.openshift.io/v1?timeout=32s: context deadline exceeded (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "authorization.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/authorization.openshift.io/v1?timeout=32s:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "build.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/build.openshift.io/v1?timeout=32s:
      context deadline exceeded (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "image.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/image.openshift.io/v1?timeout=32s:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "oauth.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/oauth.openshift.io/v1?timeout=32s:
      context deadline exceeded (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "project.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/project.openshift.io/v1?timeout=32s:
      context deadline exceeded (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "quota.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/quota.openshift.io/v1?timeout=32s:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "route.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/route.openshift.io/v1?timeout=32s:
      context deadline exceeded)
    
        APIServicesAvailable: "security.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/security.openshift.io/v1?timeout=32s:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "template.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/template.openshift.io/v1?timeout=32s:
      context deadline exceeded (Client.Timeout exceeded while awaiting headers))
    
        APIServicesAvailable: "user.openshift.io.v1" is not ready: 0 (Get https://10.205.0.1:443/apis/user.openshift.io/v1?timeout=32s:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers))'
      reason: APIServices_Error
      status: 'False'
      type: Available
    
  • Error "Client.Timeout exceeded while awaiting headers" in several pods:

    2021-01-29T03:28:34.330074737Z I0129 03:28:34.330006       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-apiserver-operator", Name:"openshift-apiserver-operator", UID:"c9ab82de-7fbe-11ea-ac8b-005056b2e4b3", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'OpenShiftAPICheckFailed' "oauth.openshift.io.v1" failed with HTTP status code 0 (Get https://10.205.0.1:443/apis/oauth.openshift.io/v1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers))
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP, OCP)
    • 4.5.16+
    • 4.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content