ROSA with Hosted Control Planes: Etcd Encryption with Customer Managed KMS Key Fails to Start Worker Nodes

Solution In Progress - Updated -

Environment

ROSA with Hosted Control Plane when usind Customer Managed KMS keys for etcd encryption.

Issue

When installing Red Hat Openshift Service on AWS (ROSA) with Hosted Control Planes and adding a Customer Managed KMS Key, the worker nodes of the cluster may fail to start if the KMS Key Policy is not correctly configured.

Note: At the time of this writing (May 29th, 2023), Hosted Control Planes is in Technology Preview and is not yet supported.

Resolution

To reslove this issue, you must do the following:

  1. Create an oidc provider and generate operator roles.
$ rosa create oidc-config --mode=auto
<snip>
I: Created OIDC provider with ARN 'arn:aws:iam::NNNNNNNNNNNN:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
<snip>
$ rosa create operator-roles --prefix <user-defined> --oidc-config-id xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
<snip>
I: Creating roles using 'arn:aws:iam::NNNNNNNNNNNN:user/<USER>'
I: Created role '<user-defined>-kube-system-capa-controller-manager' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-capa-controller-manager'
I: Created role '<user-defined>-kube-system-control-plane-operator' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-control-plane-operator'
I: Created role '<user-defined>-kube-system-kms-provider' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-kms-provider'
I: Created role '<user-defined>-openshift-image-registry-installer-cloud-credential' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-openshift-image-registry-installer-cloud-credential'
I: Created role '<user-defined>-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-openshift-ingress-operator-cloud-credentials'
I: Created role '<user-defined>-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-openshift-cluster-csi-drivers-ebs-cloud-credentials'
I: Created role '<user-defined>-openshift-cloud-network-config-controller-cloud-cre' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-openshift-cloud-network-config-controller-cloud-cre'
I: Created role '<user-defined>-kube-system-kube-controller-manager' with ARN 'arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-kube-controller-manager'
  1. Get the installer role name from the rosa command:
$ rosa list account-roles  | grep ManagedOpenShift-In
ManagedOpenShift-Installer-Role     Installer      arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Installer-Role
  1. Create a KMS key and add the following four elements to the KMS Key Policy:

Three operator roles:

  1. openshift-cluster-csi-drivers-ebs-cloud-cred
  2. kube-system-kms-provider
  3. kube-system-capa-controller-manager

One account role, the one for the installer:

  1. arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Installer-Role

In the Amazon Console, go to KMS > Customer managed keys and add a KMS key. The default key will look similar to this:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Installer-Role",
                    "arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-kms-provider",
                    "arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-openshift-cluster-csi-drivers-ebs-cloud-credentials",
                    "arn:aws:iam::NNNNNNNNNNNN:role/<user-defined>-kube-system-capa-controller-manager",
                    "arn:aws:iam::NNNNNNNNNNNN:root"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

NOTE: For more information about the managed policy definitions for those roles, refer to the policy JSON files in the following directory:

https://github.com/openshift/managed-cluster-config/tree/master/resources/sts/4.12/hypershift

Root Cause

Because the control plane in an Managed Control Plane cluster is in a different account, there must be a set of roles assigned to the KMS key policy for the Customer Managed KMS keys to be used for etcd encryption.

The documentation at the time of this writing did not provide all the steps necessary for the CMKMS keys to be used in the control plane account.

Diagnostic Steps

Steps to Reproduce the Problem

  1. Create an oidc provider and generate operator roles.
$ rosa create oidc-config --mode=auto
<snip>
I: Created OIDC provider with ARN 'arn:aws:iam::NNNNNNNNNNNN:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
<snip>
$ rosa create operator-roles --prefix <user-defined> --oidc-config-id xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  1. Create a KMS key

In the Amazon Console, go to KMS > Customer managed keys and add a KMS key. The default key will look similar to this:

{
    "Version": "2012-10-17",
    "Id": "key-default-1",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::NNNNNNNNNNNN:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}
  1. Execute an installation with the rosa CLI command, adding the three flags needed to install with etcd encrypted with a CMK KMS key:
$ export OIDC_CONFIG_ID="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$ export PRIVATE_SUBNET=$(terraform output -raw cluster-private-subnet)
$ export KMS_ARN="arn:aws:kms:us-east-1:NNNNNNNNNNNN:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

$  osa create cluster --sts \
--oidc-config-id $OIDC_CONFIG_ID \
--operator-roles-prefix <user-defined> \
--hosted-cp \
--cluster-name <user-defined> \
--region us-east-1 
--version <VERSION> \
--private-link \
--subnet-ids $PRIVATE_SUBNET \
--role-arn arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Installer-Role \
--worker-iam-role arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Worker-Role \
--support-role-arn arn:aws:iam::NNNNNNNNNNNN:role/ManagedOpenShift-Support-Role 
--etcd-encryption \
--kms-key-arn $KMS_ARN \
--etcd-encryption-kms-arn $KMS_ARN

Result

The cluster will report it is installed, but when you look at the console, or if you create a jumphost and log into the cluster via the oc command, you will see there are no workers nodes created.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments