ROSA / OSD : Custom KMS key not propagated to default storage class

Solution Verified - Updated -

Environment

This issue impacts Red Hat OpenShift Service on AWS (ROSA) or OpenShift Dedicated (on AWS) clusters deployed before 20 April 2022 that were configured to use a customer managed AWS KMS key. Clusters created after this date should not be affected.

Issue

A Red Hat OpenShift Service on AWS (ROSA) or OpenShift Dedicated (on AWS) cluster configured (at install time) to use a customer managed AWS KMS key for encryption may not use the customer supplied AWS KMS key to encrypt persistent volumes for the cluster. The specified AWS KMS key is only used to encrypt the root filesystem for cluster nodes.
Persistent volumes are still encrypted, but uses the default AWS managed KMS key for the region the cluster is in, rather than the specified customer managed KMS key.

Resolution

Summary

Remediating this issue involves creating a new storageclass - gp2-customer-kms - that will be configured with the supplied customer managed AWS KMS key ARN. Data from persistent volumes using the original gp2 storageclass which doesn’t use the supplied customer managed AWS KMS key ARN can then be migrated onto new volumes that are created with the new gp2-customer-kms storageclass.

Steps

  1. Confirm the cluster is affected by the issue using the commands above
  2. Grant access for the roles that OpenShift uses for creating PVs to use the specified AWS KMS key. This can be done either via IAM policy or directly on the AWS KMS key itself.
    • For STS clusters: ManagedOpenShift-Installer-Role, ManagedOpenShift-ControlPlane-Role, ManagedOpenShift-Support-Role, ManagedOpenShift-Worker-Role
      • Important note: If you have used custom prefixes for the STS Roles, you will need to grant access to the custom prefix versions of the roles.
    • non-STS clusters: <infrastructure-id>-master-role
      • The infrastructure ID can be found by running this command: oc get infrastructure cluster -o go-template='{{.status.infrastructureName}}{{"\n"}}'
      • For example: mycluster-xvnb-master-role
  3. Open a support case with Red Hat to implement the remediation. Red Hat SREs will create the new storageclass and recreate the persistent volumes managed by Red Hat SRE onto new PVs which use the correct gp2-customer-kms storageclass.
  4. Once completed, you can migrate data from persistent volumes using the original gp2 storageclass onto new persistent volumes that use the gp2-customer-kms storageclass.
    For more assistance on how to do this, follow the documentation located here:

Additional Resources

Migrating data between PVs

Root Cause

The default storageclass for Red Hat OpenShift Service on AWS (ROSA) or OpenShift Dedicated (on AWS) clusters did not contain a specific AWS KMS key ARN, resulting in the storageclass using the default AWS managed KMS key for the region the cluster is in.

This has been addressed for new clusters. Remediation for clusters which were created before the fix was released, requires the customer to grant the roles used by OpenShift, permission to use the specified AWS KMS key before the issue can be resolved.

Diagnostic Steps

To determine if a cluster is affected by this issue, first determine if the cluster was installed used a customer managed KMS key by running the following command:

$ oc get machine -n openshift-machine-api $(oc get infrastructure cluster -o go-template='{{.status.infrastructureName}}')-master-0 -o go-template='{{(index .spec.providerSpec.value.blockDevices 0
).ebs.kmsKey.arn}}{{"\n"}}'

If this returns an AWS KMS key ARN, proceed to the next step.
If this returns null, then the cluster was not configured to use a customer managed AWS KMS key for encryption and the cluster is unaffected by this issue.

Next, list the storageclasses on the cluster by running the following command:

$ oc describe storageclass gp2-customer-kms

Verify the output of the command includes the customer managed AWS KMS key in the parameters section. If the above command returns Error from server (NotFound): storageclasses.storage.k8s.io "gp2-customer-kms" not found or contains a different AWS KMS key ARN, then the cluster is affected by this issue.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments