ROSA STS 4.7 clusters are prevented from upgrading with an annotation on the CloudCredential CustomResource

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4.7
  • AWS security token service (STS)

Issue

  • To ensure AWS policies have been updated before enabling ROSA STS 4.7 cluster upgrades, they will require annotation on the CloudCredential CR, by the cluster administrator.
  • ROSA STS clusters upgrading from 4.7 to 4.8 need the below steps to ensure stability of the cluster through and after the upgrade operation.

  • There are warnings in a ROSA STS 4.7 cluster:

    One or more cluster operators have been blocking minor version cluster upgrades for at least an hour for reason MissingUpgradeableAnnotation. For more information refer to https://console-openshift-console.apps.[cluster_name].[xxxx].p1.openshiftapps.com/settings/cluster

    This cluster should not be updated to the next minor version.
Cluster operator cloud-credential cannot be upgraded between minor versions: Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade.

Resolution

This KCS only applies to ROSA 4.7 clusters with STS. For upgrading ROSA clusters from 4.8 to 4.9, please refer to KCS 6627971: Error while trying to upgrade ROSA cluster to 4.9. For other releases, please refer to OSD/ROSA cluster requires user action before install or upgrade.

Note: A minor OCP 4 version is the x in 4.x.z, and the z represents the patch version. So, upgrading from 4.7.z to 4.7.z+n is still possible while this message is shown.

Upgrade from ROSA STS 4.7 to 4.8

To upgrade from ROSA STS 4.7 to 4.8, the following tasks need to be done:

  1. It's needed to upgrade to latest/current ROSA CLI (1.1.3 or newer).

  2. The openshift-machine-api-operator requires the elasticloadbalancing:DeregisterTargets permission for 4.8. As such, a 4.7 cluster needs the new policy setup.
    The AWS account owner for the cluster to upgrade must execute this to enable the necessary AWS permissions, and include any prefix used (if used) when the account roles were created, with --prefix:

    $ rosa create account-roles

  3. As cluster-admin execute the following on the 4.7 cluster. This annotation indicates the cluster is ready to upgrade to 4.8:

    $ oc annotate cloudcredential cluster cloudcredential.openshift.io/upgradeable-to="v4.8"

  4. Proceed with upgrading your cluster using ROSA CLI or the Red Hat Cloud Console.

Root Cause

Additional permissions are required for ROSA 4.8 that were part of a different STS role in ROSA 4.7.

Note: A minor OCP 4 version is the x in 4.x.z, and the z represents the patch version. So, upgrading from 4.7.z to 4.7.z+n is still possible while this message is shown.

Diagnostic Steps

Check in the clusterversion for the following message:

$ oc get clusterversion version -o json
[...]
                "message": "Cluster operator cloud-credential cannot be upgraded between minor versions: Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade.",
                "reason": "MissingUpgradeableAnnotation",
                "status": "False",
                "type": "UpgradeableClusterOperators"
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments