OCM 3055 - Your ROSA Cluster's Installation Role does not have permissions to use the default KMS Key in your AWS Account

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA 4.x)

Issue

When installing a ROSA cluster for the first time, you may notice some KMS key errors. This could either be invalidKMSKey or InvalidKMSKey.InvalidState.

The error code being OCM 3055

You would also notice errors corresponding to the logs below:

Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

If you are using the RHCS Terraform Providers, you may notice something similar to this then:

Your cluster's installation role does not have permissions to use the default KMS key in your AWS account. Please ensure that the installation role has permissions to use this key and try again.

Resolution

  1. Start by checking if EBS encryption is enabled in the EC2 settings. You can check the status for that by following the steps here:
    AWS Check EBS Encryption

  2. Check to see if the AWS Specified Key is enabled in there and not an invalidKMSKey that doesn't exist. This could happen when an old key was specified and deleted but EBS did not fall back to another key

  3. If none of this work, disable EBS encryption entirely. If this is still a requirement you can't disable. You can specify a customer-managed-key during ROSA install following the steps here
    Install ROSA Cluster with Custom KMS Key

Root Cause

By default there are KMS keys assigned by AWS for EBS encryption. In a customer's environment there maybe some changes that require a custom KMS key and this implementation doesn't change even if the KMS key is deleted. EBS continues to point to the deleted key. ROSA Installation would thus fail since it would be pointing to an invalidKey

EBS encryption using a default key specified by AWS could also cause an issue, since ROSA may not have the enough privileges to assume the role on this key. Thus as a final resort its recommended to disable EBS Encryption and use a customer-managed-key to encrypt the volumes instead following the steps above

Diagnostic Steps

  1. To determine if a cluster is affected by this issue, first check the cluster installation logs for a KMS key error. That can be achieved using the following commands:
    rosa logs install -c cluster-name

  2. You could double confirm by using the search tool on your terminal and search for KMS key errors.

  3. You could also check the OCM UI, under installation issues and notice the code that the error throws. In this case it would be OCM-3055

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments