ROSA 4.11 clusters require user action before scheduling a minor version upgrade

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4.11
    • AWS security token service (STS)

Issue

  • Red Hat OpenShift Service on AWS (ROSA) cluster upgrading from 4.11 to 4.12 need cluster owner action to ensure stability of the cluster through and after the upgrade operation.
  • To ensure that cluster owners understand the changes occurring as as part of upgrading the underlying Kubernetes to v1.25, there are acknowledgements necessary to ensure cluster owners have taken necessary precautions.
  • To ensure AWS policies have been updated before enabling ROSA STS 4.11 cluster upgrades to 4.12, AWS role policy updates are part of the upgrade workflow outlined in the Resolution below.

Resolution

Note: For other releases, please refer to OSD/ROSA cluster requires user action before install or upgrade.

Note: For ROSA IAM (non-STS) clusters, the STS role/policy related steps in this guide do not apply; those clusters will not have a step in the CLI workflow about ensuring policies are updated.

Prerequisites

Before starting with the upgrade to OpenShift 4.12 (which uses Kubernetes 1.25), as a significant number of deprecated APIs are removed in that version, it's necessary to review KCS 6955381: Preparing to upgrade to OpenShift Container Platform 4.12 to understand the changes and to verify if those changes could affect the custom applications in the cluster.

Prepare to Upgrade ROSA STS

To prepare for upgrading ROSA STS from from 4.11 to 4.12, the following tasks need to be done:

  1. Upgrade to latest/current ROSA CLI.

    As part of the upgrade task, the ROSA CLI helps you by updating STS policies for you with 'auto' mode, or allow you to inspect the changes before applying them, with 'manual' mode.

  2. Understand upgrading ROSA clusters with STS using ROSA CLI or the Red Hat Cloud Console.

  3. Prepare for an upgrade by updating your ROSA STS roles/policies to latest. List your account-roles and notice they are grouped by the prefix in their naming scheme. Using the prefix that is relevant to the roles for your cluster (assuming you're logged in) you can upgrade each group as shown here:

    $ rosa list account-roles
ROLE NAME                           ROLE TYPE      ROLE ARN                                                           OPENSHIFT VERSION
customprefix-ControlPlane-Role      Control plane  arn:aws:iam::00sample0000:role/customprefix-ControlPlane-Role      4.11
customprefix-Installer-Role         Installer      arn:aws:iam::00sample0000:role/customprefix-Installer-Role         4.11
customprefix-Support-Role           Support        arn:aws:iam::00sample0000:role/customprefix-Support-Role           4.11
customprefix-Worker-Role            Worker         arn:aws:iam::00sample0000:role/customprefix-Worker-Role            4.11
ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.11
ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.11
ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.11
ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.11

$ rosa list clusters 

$ rosa upgrade account-roles --prefix ManagedOpenShift

$ rosa upgrade operator-roles -c <Cluster ID>

$ rosa list account-roles |grep ManagedOpenShift
ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.12
ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.12
ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.12
ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.12

$ rosa upgrade cluster -c <cluster-name>
#<this task proceeds depending on the state of your selected cluster>

    And now your ROSA cluster and STS account roles are prepared for and updated to the 4.12 release. There is no harm in upgrading the account roles to a higher version like 4.13 as account roles are backward compatible.

Optional:

If you wish to continue at the CLI to initiate your upgrade, as a cluster owner, proceed with the following step.
The upgrade cluster function checks for your acknowledgement of preparedness for Kubernetes 1.25 API deprecation details. After these checks are done, you are offered to schedule your upgrade to the 4.12 version for the cluster.
If you have more than one cluster, you may need to list them to determine the cluster you wish to upgrade. You may wish to list your clusters for the cluster ID and then run the upgrade command on the 4.11 cluster (the rosa upgrade cluster command will allow you to schedule an upgrade interactively):

$ rosa list clusters
$ rosa upgrade cluster -c <cluster id>

Root Cause

ROSA clusters upgrading from 4.11 to 4.12 need cluster owner action to ensure stability of the cluster through and after the upgrade operation.

For additional information about Kubernetes deprecated APIs, refer to:

For ROSA STS Clusters, AWS STS role policies required for ROSA 4.12 must be up to date when coming from older versions.

Diagnostic Steps

Check the version of ROSA CLI:

$ rosa version
1.2.9

Check if the cluster is version 4.11:

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.zz    True        False         8d      Cluster version is 4.11.zz

or

rosa describe cluster -c <cluster id> |grep Version
OpenShift Version:          4.11.zz

Check your cluster version from the cluster settings or cluster overview at https://console.redhat.com/openshift.

To review details of role and policies necessary for ROSA STS, you may review all policies by generating them locally with:

$ rosa create account-roles --mode manual

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments