ROSA STS requires user action before install or upgrade

Solution Verified - Updated -

Environment

For New Clusters:
- Red Hat OpenShift Service on AWS (ROSA) 4.10+
- AWS security token service (STS)

For Upgrading Clusters:
- Red Hat OpenShift Service on AWS (ROSA) 4.8+
- AWS security token service (STS)

ROSA CLI
- versions older than 1.1.12

Issue

  • Before installing Red Hat OpenShift Service on AWS (ROSA) 4.10 or upgrading ROSA clusters to the next minor version (y-stream), cluster owner action is necessary to have the required STS account-roles/policies.
  • To ensure AWS policies have been updated before enabling ROSA STS cluster installs (or upgrades from an older to a newer minor version), AWS role policy updates are part of the upgrade workflow outlined in the Resolution below.

Or similarly:

User tries to install a cluster and even if account roles already exist, this warning is shown at ROSA CLI (versions older than 1.1.12):

W: No account roles found. You will need to manually set them in the next steps or run 'rosa create account-roles' to create them first.

Resolution

Note: For upgrades from OCP 4.7 to 4.8, please refer to: ROSA STS 4.7 clusters are prevented from upgrading with an annotation on the CloudCredential CustomResource.

Note: For OSD 4.8 upgrades, please refer to: OSD 4.8 clusters require user action before scheduling a minor version upgrade.

Note: For ROSA IAM (non-STS) clusters, the STS role/policy related steps in this guide do not apply; those clusters will not have a step in the CLI workflow about ensuring policies are updated.

Note: For ROSA STS 4.8 to 4.9 upgrades, please refer to: ROSA 4.8 clusters require user action before scheduling a minor version upgrade

Prepare the ROSA STS account role policies

To prepare for installing ROSA STS 4.10+ or upgrading ROSA STS from from a minor version to the next minor version (ex: 4.9 to 4.10), the following tasks need to be done:

  1. Upgrade to latest/current ROSA CLI
    Must be 1.1.12 or newer, but latest as available.

    As part of this task, the ROSA CLI can update STS policies for you with 'auto' mode, or allow you to inspect the changes before applying them, with 'manual' mode.

  2. Understand upgrading ROSA clusters with STS using ROSA CLI or the Red Hat Cloud Console.

  3. Prepare for an install or upgrade by updating your ROSA STS roles/policies to latest. List your account-roles and notice they are grouped by the prefix in their naming scheme. Using the prefix that is relevant to the roles for your cluster (assuming you're logged in) you can upgrade each group as shown in the example here:

    $ rosa list account-roles
    ROLE NAME                           ROLE TYPE      ROLE ARN                                                           OPENSHIFT VERSION
    customprefix-ControlPlane-Role      Control plane  arn:aws:iam::00sample0000:role/customprefix-ControlPlane-Role      4.9
    customprefix-Installer-Role         Installer      arn:aws:iam::00sample0000:role/customprefix-Installer-Role         4.9
    customprefix-Support-Role           Support        arn:aws:iam::00sample0000:role/customprefix-Support-Role           4.9
    customprefix-Worker-Role            Worker         arn:aws:iam::00sample0000:role/customprefix-Worker-Role            4.9
    ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.8
    ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.8
    ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.8
    ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.8
    
    $ rosa upgrade account-roles --prefix ManagedOpenShift
    #<this task proceeds depending on the state of your selected account-roles>
    
    $ rosa list account-roles |grep ManagedOpenShift
    ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.10
    ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.10
    ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.10
    ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.10
    

    In the above example, ROSA STS account roles are prepared for the 4.10 release.

  4. Ensure your operator-roles are up to date as well

    $ #rosa upgrade operator-roles -c <cluster name> --version <target/new OpenShift minor version>
    $ rosa upgrade operator-roles -c my-cluster --version 4.10
    

    With account-roles at the appropriate 'OpenShift Version', and operator roles updated, you may now proceed to schedule an upgrade in the OCM UI or ROSA CLI.

Optional (upgrade):

If you wish to continue at the CLI to initiate your upgrade (assuming an upgrade edge is available), as a cluster owner, proceed with the following step.
The upgrade cluster function checks for your acknowledgement of preparedness for any notable deprecation warnings. After these checks are done, you are offered to schedule your upgrade to the next minor version for the cluster.
If you have more than one cluster, you may need to list them to determine the cluster you wish to upgrade. You may wish to list your clusters for the cluster ID and then run the upgrade command on the appropriate cluster (the rosa upgrade cluster command will allow you to schedule an upgrade interactively):

$ rosa list clusters
$ rosa upgrade cluster -c <cluster id>

Optional (new cluster creation):

If you wish to continue at the CLI to initiate a cluster creation, please continue with the official documentation.

Root Cause

To create new ROSA STS clusters, the latest ROSA CLI is necessary in order to have the requisite STS account roles/policies. Must be 1.1.12 or newer, but latest as available.

ROSA STS clusters upgrading from older to newer minor versions require the latest ROSA CLI (at least 1.1.12) to update STS account roles/policies.

Diagnostic Steps

Check the version of ROSA CLI:

$ rosa version
1.2.2

Check the cluster version:

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.xx    True        False         8d      Cluster version is 4.8.xx

or

rosa describe cluster -c <cluster id> |grep Version
OpenShift Version:          4.8.32

Check your cluster version from the cluster settings or cluster overview at https://console.redhat.com/openshift.

To review details of role and policies necessary for ROSA STS, you may review all policies by generating them locally with:

$ rosa create account-roles --mode manual

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments