openshift-apiserver cluster-operator goes unavailable during cluster installation or upgrade from OCP 4.6 to 4.7

Solution Unverified - Updated -

Issue

  • After deploying OCP 4.7 or upgrading OCP cluster from 4.6 to 4.7, multiple clusteroperator goes into the Degraded state.

    authentication                            4.7.0    False      True         True      4d
    console                                   4.7.0    False      False        True      2h20m
    image-registry                            4.7.0    False      True         True      1d
    ingress                                   4.7.0    True       False        True      6h41m
    monitoring                                4.7.0    False      True         True      1m19s
    network                                   4.7.0    True       False        False     3h16m
    openshift-apiserver                       4.7.0    False      False        False     4d
    operator-lifecycle-manager-packageserver  4.7.0    False      True         False     2s
    
  • openshift-apiserver cluster-operator shows below logs:

    $ oc get co openshift-apiserver -o yaml | grep "not ready" -A1
    message: 'APIServicesAvailable: "apps.openshift.io.v1" is not ready: 503 (the
      server is currently unable to handle the request)
      APIServicesAvailable: "authorization.openshift.io.v1" is not ready: 503 (the
      server is currently unable to handle the request)
      APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)
      APIServicesAvailable: "image.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)
      APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)
      APIServicesAvailable: "route.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)'
    
  • Cluster upgrade is stuck at 1% complete:

    $ oc get clusterversion
    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.6.18    True        True          4d22h   Working towards 4.7.0: 8 of 668 done (1% complete)
    $ oc get clusterversion
    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.6.18    True        True          4d22h   Unable to apply 4.7.0: the control plane is reporting an internal error
    

Environment

  • Red Hat OpenShift Container Platform [RHOCP]
    • Upgrade from 4.6 to 4.7
    • 4.7 (Fresh install)
  • VMWare Vsphere ESXi
    • 6.7, 7.0
  • VM hardware versions 14 and 15. Additional hardware versions beyond version 13 may be impacted.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content