ClusterNotUpgradeable and AdminAckRequired in OCP 4.8

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (RHOCP)
    • 4.8

Issue

  • A ClusterNotUpgradeable alert is shown in an OCP 4.8 cluster, with reason AdminAckRequired:

    One or more cluster operators have been blocking minor version cluster upgrades for at least an hour for reason AdminAckRequired.

  • Trying to upgrade from OCP 4.8 to OCP 4.9 fails with an error:

    Upgradeable=False

    Reason: AdminAckRequired
Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.

Resolution

The OpenShift Container Platform 4.9 is based on Kubernetes 1.22 and that version removed a significant number of deprecated v1beta1 APIs.

OCP 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OCP 4.8 to 4.9, and alerts are shown in the cluster. This manual acknowledgment is to help prevent issues during and after the upgrade to OCP 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster.

Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

To know more about these removed Kubernetes APIs and provide the administrator acknowledgment after evaluating your cluster for removed APIs, refer to the article: Preparing to upgrade to OpenShift Container Platform 4.9.

Root Cause

The OpenShift Container Platform 4.9 is based on Kubernetes 1.22 and that version removed a significant number of deprecated v1beta1 APIs.

All OCP 4.8 clusters require an administrator acknowledgment before they can be upgraded to OCP 4.9.

Diagnostic Steps

Check the fired alerts in the cluster using the cli for the message One or more cluster operators have been blocking minor version cluster upgrades for at least an hour for reason AdminAckRequired:

$ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring -o jsonpath='{@.spec.host}')
$ curl -k -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)"  https://$ALERT_MANAGER/api/v1/alerts
[...]
        "message": "One or more cluster operators have been blocking minor version cluster upgrades for at least an hour for reason AdminAckRequired. [...]
[...]

The oc adm upgrade command fails:

$ oc adm upgrade
Cluster version is 4.8.x

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
  the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments