ROSA STS requires user action before install or upgrade

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • New 4.10+ clusters
    • For upgrading 4.8+ clusters
  • AWS security token service (STS)
  • rosa CLI
    • 1.1.12+

Issue

  • Before installing Red Hat OpenShift Service on AWS (ROSA) 4.10 or upgrading ROSA clusters to the next minor version (y-stream), cluster owner action is necessary to have the required STS account-roles/policies.
  • To ensure AWS policies have been updated before enabling ROSA STS cluster installs (or upgrades from an older to a newer minor version), AWS role policy updates are part of the upgrade workflow outlined in the Resolution below.

  • User tries to install a cluster and even if account roles already exist, this warning is shown at rosa CLI (versions older than 1.1.12):

    W: No account roles found. You will need to manually set them in the next steps or run 'rosa create account-roles' to create them first.

Resolution

Note: For other releases, please refer to OSD/ROSA cluster requires user action before install or upgrade.

Note: For ROSA IAM (non-STS) clusters, the STS role/policy related steps in this guide do not apply; those clusters will not have a step in the CLI workflow about ensuring policies are updated.

Prepare the ROSA STS account role policies

To prepare for installing ROSA STS 4.10+ or upgrading ROSA STS from from a minor version to the next minor version (ex: 4.9 to 4.10), the following tasks need to be done:

  1. Upgrade the rosa CLI
    Must be latest available version (1.2.11 at the time of writing).
    As part of this task, the rosa CLI can update STS policies automatically with 'auto' mode, or allow to inspect the changes before applying them, with 'manual' mode.

  2. Understand upgrading ROSA clusters with STS using rosa CLI or the Red Hat Cloud Console.

  3. Prepare for an install or upgrade by updating the ROSA STS roles/policies to latest. List the account-roles and notice they are grouped by the prefix in their naming scheme. Using the prefix that is relevant to the roles for the cluster (it's needed to be logged in) it's possible to upgrade each group as shown in the example here:

    $ rosa list account-roles
ROLE NAME                           ROLE TYPE      ROLE ARN                                                           OPENSHIFT VERSION
customprefix-ControlPlane-Role      Control plane  arn:aws:iam::00sample0000:role/customprefix-ControlPlane-Role      4.9
customprefix-Installer-Role         Installer      arn:aws:iam::00sample0000:role/customprefix-Installer-Role         4.9
customprefix-Support-Role           Support        arn:aws:iam::00sample0000:role/customprefix-Support-Role           4.9
customprefix-Worker-Role            Worker         arn:aws:iam::00sample0000:role/customprefix-Worker-Role            4.9
ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.8
ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.8
ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.8
ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.8

$ rosa upgrade account-roles --prefix ManagedOpenShift
#<this task proceeds depending on the state of the selected account-roles>

$ rosa list account-roles |grep ManagedOpenShift
ManagedOpenShift-ControlPlane-Role  Control plane  arn:aws:iam::00sample0000:role/ManagedOpenShift-ControlPlane-Role  4.10
ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::00sample0000:role/ManagedOpenShift-Installer-Role     4.10
ManagedOpenShift-Support-Role       Support        arn:aws:iam::00sample0000:role/ManagedOpenShift-Support-Role       4.10
ManagedOpenShift-Worker-Role        Worker         arn:aws:iam::00sample0000:role/ManagedOpenShift-Worker-Role        4.10

    In the above example, ROSA STS account roles are prepared for the 4.10 release.

  4. Ensure that operator-roles are up to date as well:

    $ rosa upgrade operator-roles -c my-cluster --version 4.10.z

    Note: To upgrade from 4.9.z to 4.10.z the flag --version 4.10.z is needed. For upgrading from 4.10.z to 4.11.z, the --version should not be included in the command)

With account-roles at the appropriate OpenShift Version, and operator-roles updated, it's possible to proceed to schedule an upgrade in the OCM UI or rosa CLI.

Optional (for upgrading):

To continue at the CLI to initiate the upgrade (assuming an upgrade edge is available), as a cluster owner, proceed with the following step.
The upgrade cluster function checks for the acknowledgement of preparedness for any notable deprecation warnings. After these checks are done, it's offered to schedule the upgrade to the next minor version for the cluster.
If there is more than one cluster, it's needed to list them to determine the cluster to be upgraded. It's possible to list the clusters for the cluster ID and then run the upgrade command on the appropriate cluster (the rosa upgrade cluster command will allow to schedule an upgrade interactively):

$ rosa list clusters
$ rosa upgrade cluster -c <cluster id>

Optional (for new cluster creation):

To continue at the CLI to initiate a cluster creation, please continue with the official documentation.

Root Cause

To create new ROSA STS clusters, the latest rosa CLI is necessary in order to have the requisite STS account roles/policies. Must be latest available version (1.2.11 at the time of writing).

ROSA STS clusters upgrading from older to newer minor versions require the latest rosa CLI (at least 1.2.11) to update STS account roles/policies.

Diagnostic Steps

Check the version of rosa CLI:

$ rosa version
1.2.11

Check the cluster version:

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.xx    True        False         8d      Cluster version is 4.10.xx

or

$ rosa describe cluster -c <cluster id> |grep Version
OpenShift Version:          4.10.xx

Check the cluster version from the cluster settings or cluster overview at https://console.redhat.com/openshift.

To review details of role and policies necessary for ROSA STS, check all policies by generating them locally with:

$ rosa create account-roles --mode manual

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments