Why is ClusterNotUpgradeable firing after updating to RHOCP 4.8?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.8
  • Red Hat OpenShift GitOps
    • 1.2
  • Operators

Issue

  • After upgrading a cluster to RHOCP 4.8, a ClusterNotUpgradeable alert is seen. However the upgrade seams to be finished successful and there is no error reported.
  • Why the alert ClusterNotUpgradeable is firing in OpenShift 4.8 clusters that have OpenShift GitOps installed?

Resolution

  • Some operators could be not compatible with OpenShift 4.9, so the upgrade to that version will be denied. But the upgrades to newer 4.8.z are still possible and working. Refer to the Diagnostic Steps section for additional information.

Note: If the alert includes the AdminAckRequired reason, refer to ClusterNotUpgradeable and AdminAckRequired in OCP 4.8.

  • For example, Red Hat OpenShift GitOps 1.2 is not compatible with Red Hat OpenShift Container Platform 4.9, which is why the IncompatibleOperatorsInstalled condition in the Cluster Version Operator (CVO) is set and thus triggering ClusterNotUpgradeable which is looking for IncompatibleOperatorsInstalled being set.

Root Cause

Some operators could be not compatible with OpenShift 4.9, so the upgrade to that version will be denied.

Diagnostic Steps

The Upgradeable condition in the Cluster Version Operator (CVO) should contain more details why IncompatibleOperatorsInstalled is set. To extract those details, run oc get clusterversion version -o json:

    {
        "apiVersion": "config.openshift.io/v1",
        "kind": "ClusterVersion",
        [...]
        "spec": {
            "channel": "stable-4.8",
            "clusterID": "XXXXXXXX-XXXX-XXXX-XXXXXXXX",
            "desiredUpdate": {
                "image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
                "version": "4.8.10"
            },
            "upstream": "https://api.openshift.com/api/upgrades_info/v1/graph"
        },
        "status": {
            "availableUpdates": null,
            "conditions": [
                [...]
                {
                    "lastTransitionTime": "2021-09-13T07:32:56Z",
                    "message": "Cluster operator operator-lifecycle-manager cannot be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators/openshift-gitops-operator.v1.2.0 is incompatible with OpenShift minor versions greater than 4.8",  <-- This shows details why IncompatibleOperatorsInstalled is set
                    "reason": "IncompatibleOperatorsInstalled",
                    "status": "False",
                    "type": "Upgradeable"
                }
                [...]
            ],
            [...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments