Chapter 18. Managing cloud provider credentials

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • GCP: GCPProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }
      ...

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \ 1
        -n <secret_namespace> 2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

7. To verify that the credentials have changed:

   1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.
   2. Verify that the contents of the Value field or fields are different than the previously recorded information.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   GCP	gcp-credentials

3. Click the Options menu in the same row as the secret and select Delete Secret.

Procedure

1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.

2. In the table on the Secrets page, find the root secret for your cloud provider.

   Platform	Secret name
   AWS	aws-creds
   Azure	azure-credentials
   GCP	gcp-credentials
   RHOSP	openstack-credentials
   RHV	ovirt-credentials
   vSphere	vsphere-creds

3. Click the Options menu in the same row as the secret and select Edit Secret.
4. Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials.
5. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save.

6. If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects.

   1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

   2. Get the names and namespaces of all referenced component secrets:

      $ oc -n openshift-cloud-credential-operator get CredentialsRequest \
        -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'

      where <provider_spec> is the corresponding value for your cloud provider:

      • AWS: AWSProviderSpec
      • Azure: AzureProviderSpec
      • GCP: GCPProviderSpec
      • RHOSP: OpenStackProviderSpec
      • RHV: OvirtProviderSpec
      • vSphere: VSphereProviderSpec

      Partial example output for AWS

      {
        "name": "ebs-cloud-credentials",
        "namespace": "openshift-cluster-csi-drivers"
      }
      {
        "name": "cloud-credential-operator-iam-ro-creds",
        "namespace": "openshift-cloud-credential-operator"
      }
      ...

   3. Delete each of the referenced component secrets:

      $ oc delete secret <secret_name> \ 1
        -n <secret_namespace> 2

      1

      Specify the name of a secret.

      2

      Specify the namespace that contains the secret.

      Example deletion of an AWS secret

      $ oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

      You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones.

7. To verify that the credentials have changed:

   1. In the Administrator perspective of the web console, navigate to Workloads → Secrets.
   2. Verify that the contents of the Value field or fields are different than the previously recorded information.

Procedure

1. Generate a private key to sign the ServiceAccount object:

   $ openssl genrsa -out sa-signer 4096

2. Generate a ServiceAccount object public key:

   $ openssl rsa -in sa-signer -pubout -out sa-signer.pub

3. Create an S3 bucket to hold the OIDC configuration:

   $ aws s3api create-bucket --bucket <oidc_bucket_name> --region <aws_region> --create-bucket-configuration LocationConstraint=<aws_region>

   Note

   If the value of <aws_region> is us-east-1, do not specify the LocationConstraint parameter.

4. Retain the S3 bucket URL:

   OPENID_BUCKET_URL="https://<oidc_bucket_name>.s3.<aws_region>.amazonaws.com"

5. Build an OIDC configuration:

   1. Create a file named keys.json that contains the following information:

      {
          "keys": [
              {
                  "use": "sig",
                  "kty": "RSA",
                  "kid": "<public_signing_key_id>",
                  "alg": "RS256",
                  "n": "<public_signing_key_modulus>",
                  "e": "<public_signing_key_exponent>"
              }
          ]
      }

      Where:

      • <public_signing_key_id> is generated from the public key with:

        $ openssl rsa -in sa-signer.pub -pubin --outform DER | openssl dgst -binary -sha256 | openssl base64 | tr '/+' '_-' | tr -d '='

        This command converts the public key to DER format, performs a SHA-256 checksum on the binary representation, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.

      • <public_signing_key_modulus> is generated from the public key with:

        $ openssl rsa -pubin -in sa-signer.pub -modulus -noout | sed  -e 's/Modulus=//' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='

        This command prints the modulus of the public key, extracts the hex representation of the modulus, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to base64URL encoding.

      • <public_signing_key_exponent> is generated from the public key with:

        $ printf "%016x" $(openssl rsa -pubin -in sa-signer.pub -noout -text | grep Exponent | awk '{ print $2 }') |  awk '{ sub(/(00)+/, "", $1); print $1 }' | xxd -r -p | base64 -w0 | tr '/+' '_-' | tr -d '='

        This command extracts the decimal representation of the public key exponent, prints it as hex with a padded 0 if needed, removes leading 00 pairs, converts the ASCII hex to binary, encodes the data with base64 encoding, and then changes the base64-encoded output to use only characters that can be used in a URL.

   2. Create a file named openid-configuration that contains the following information:

      {
      	"issuer": "$OPENID_BUCKET_URL",
      	"jwks_uri": "${OPENID_BUCKET_URL}/keys.json",
          "response_types_supported": [
              "id_token"
          ],
          "subject_types_supported": [
              "public"
          ],
          "id_token_signing_alg_values_supported": [
              "RS256"
          ],
          "claims_supported": [
              "aud",
              "exp",
              "sub",
              "iat",
              "iss",
              "sub"
          ]
      }

6. Upload the OIDC configuration:

   $ aws s3api put-object --bucket <oidc_bucket_name> --key keys.json --body ./keys.json

   $ aws s3api put-object --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --body ./openid-configuration

   Where <oidc_bucket_name> is the S3 bucket that was created to hold the OIDC configuration.

7. Allow the AWS IAM OpenID Connect (OIDC) identity provider to read these files:

   $ aws s3api put-object-acl --bucket <oidc_bucket_name> --key keys.json --acl public-read

   $ aws s3api put-object-acl --bucket <oidc_bucket_name> --key '.well-known/openid-configuration' --acl public-read

8. Create an AWS IAM OIDC identity provider:

   1. Get the certificate chain from the server that hosts the OIDC configuration:

      $ echo | openssl s_client -servername $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com -connect $<oidc_bucket_name>.s3.$<aws_region>.amazonaws.com:443 -showcerts 2>/dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; out="cert"a".pem"; print >out}'

   2. Calculate the fingerprint for the certificate at the root of the chain:

      $ export BUCKET_FINGERPRINT=$(openssl x509 -in cert<number>.pem -fingerprint -noout | sed -e 's/.*Fingerprint=//' -e 's/://g')

      Where <number> is the highest number in the files that were saved. For example, if 2 is the highest number in the files that were saved, use cert2.pem.

   3. Create the identity provider:

      $ aws iam create-open-id-connect-provider --url $OPENID_BUCKET_URL --thumbprint-list $BUCKET_FINGERPRINT --client-id-list openshift sts.amazonaws.com

   4. Retain the returned ARN of the newly created identity provider. This ARN is later referred to as <aws_iam_openid_arn>.

9. Generate IAM roles:

   1. Locate all CredentialsRequest CRs in this release image that target the cloud you are deploying on:

      $ oc adm release extract quay.io/openshift-release-dev/ocp-release:4.<y>.<z>-x86_64 --credentials-requests --cloud=aws

      Where <y> and <z> are the numbers corresponding to the version of OpenShift Container Platform you are installing.

   2. For each CredentialsRequest CR, create an IAM role of type Web identity using the previously created IAM Identity Provider that grants the necessary permissions and establishes a trust relationship that trusts the identity provider previously created.

      For example, for the openshift-machine-api-operator CredentialsRequest CR in 0000_30_machine-api-operator_00_credentials-request.yaml, create an IAM role that allows an identity from the created OIDC provider created for the cluster, similar to the following:

      {
          "Role": {
              "Path": "/",
              "RoleName": "openshift-machine-api-aws-cloud-credentials",
              "RoleId": "ARSOMEROLEID",
              "Arn": "arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials",
              "CreateDate": "2021-01-06T15:54:13Z",
              "AssumeRolePolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                      {
                          "Effect": "Allow",
                          "Principal": {
                              "Federated": "<aws_iam_openid_arn>"
                          },
                          "Action": "sts:AssumeRoleWithWebIdentity",
                          "Condition": {
                              "StringEquals": {
                                  "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com/$BUCKET_NAME:aud": "openshift"
                              }
                          }
                      }
                  ]
              },
              "Description": "OpenShift role for openshift-machine-api/aws-cloud-credentials",
              "MaxSessionDuration": 3600,
              "RoleLastUsed": {
                  "LastUsedDate": "2021-02-03T02:51:24Z",
                  "Region": "<aws_region>"
              }
          }
      }

      Where <aws_iam_openid_arn> is the returned ARN of the newly created identity provider.

   3. To further restrict the role such that only specific cluster ServiceAccount objects can assume the role, modify the trust relationship of each role by updating the .Role.AssumeRolePolicyDocument.Statement[].Condition field to the specific ServiceAccount objects for each component.

      • Modify the trust relationship of the cluster-image-registry-operator role to have the following condition:

        "Condition": {
          "StringEquals": {
            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
              "system:serviceaccount:openshift-image-registry:registry",
              "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator"
            ]
          }
        }

      • Modify the trust relationship of the openshift-ingress-operator to have the following condition:

        "Condition": {
          "StringEquals": {
            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
              "system:serviceaccount:openshift-ingress-operator:ingress-operator"
            ]
          }
        }

      • Modify the trust relationship of the openshift-cluster-csi-drivers to have the following condition:

        "Condition": {
          "StringEquals": {
            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
              "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-operator",
              "system:serviceaccount:openshift-cluster-csi-drivers:aws-ebs-csi-driver-controller-sa"
            ]
          }
        }

      • Modify the trust relationship of the openshift-machine-api to have the following condition:

        "Condition": {
          "StringEquals": {
            "<oidc_bucket_name>.s3.<aws_region>.amazonaws.com:sub": [
              "system:serviceaccount:openshift-machine-api:machine-api-controllers"
            ]
          }
        }

10. For each IAM role, attach an IAM policy to the role that reflects the required permissions from the corresponding CredentialsRequest objects.

    For example, for openshift-machine-api, attach an IAM policy similar to the following:

    {
        "RoleName": "openshift-machine-api-aws-cloud-credentials",
        "PolicyName": "openshift-machine-api-aws-cloud-credentials",
        "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "ec2:CreateTags",
                        "ec2:DescribeAvailabilityZones",
                        "ec2:DescribeDhcpOptions",
                        "ec2:DescribeImages",
                        "ec2:DescribeInstances",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeSubnets",
                        "ec2:DescribeVpcs",
                        "ec2:RunInstances",
                        "ec2:TerminateInstances",
                        "elasticloadbalancing:DescribeLoadBalancers",
                        "elasticloadbalancing:DescribeTargetGroups",
                        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                        "elasticloadbalancing:RegisterTargets",
                        "iam:PassRole",
                        "iam:CreateServiceLinkedRole"
                    ],
                    "Resource": "*"
                },
                {
                    "Effect": "Allow",
                    "Action": [
                        "kms:Decrypt",
                        "kms:Encrypt",
                        "kms:GenerateDataKey",
                        "kms:GenerateDataKeyWithoutPlainText",
                        "kms:DescribeKey"
                    ],
                    "Resource": "*"
                },
                {
                    "Effect": "Allow",
                    "Action": [
                        "kms:RevokeGrant",
                        "kms:CreateGrant",
                        "kms:ListGrants"
                    ],
                    "Resource": "*",
                    "Condition": {
                        "Bool": {
                            "kms:GrantIsForAWSResource": true
                        }
                    }
                }
            ]
        }
    }

11. Prepare to run the OpenShift Container Platform installer:

    1. Create the install-config.yaml file:

       $ ./openshift-install create install-config

    2. Configure the cluster to install with the CCO in manual mode:

       $ echo "credentialsMode: Manual" >> install-config.yaml

    3. Create install manifests:

       $ ./openshift-install create manifests

    4. Create a tls directory, and copy the private key generated previously there:

       Note

       The target file name must be ./tls/bound-service-account-signing-key.key.

       $ mkdir tls ; cp <path_to_service_account_signer> ./tls/bound-service-account-signing-key.key

    5. Create a custom Authentication CR with the file name cluster-authentication-02-config.yaml:

       $ cat << EOF > manifests/cluster-authentication-02-config.yaml
       apiVersion: config.openshift.io/v1
       kind: Authentication
       metadata:
         name: cluster
       spec:
         serviceAccountIssuer: $OPENID_BUCKET_URL
       EOF

    6. For each CredentialsRequest CR that is extracted from the release image, create a secret with the target namespace and target name that is indicated in each CredentialsRequest, substituting the AWS IAM role ARN created previously for each component:

       Example secret manifest for openshift-machine-api:

       $ cat manifests/openshift-machine-api-aws-cloud-credentials-credentials.yaml
       apiVersion: v1
       stringData:
         credentials: |-
           [default]
           role_arn = arn:aws:iam::123456789012:role/openshift-machine-api-aws-cloud-credentials
           web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
       kind: Secret
       metadata:
         name: aws-cloud-credentials
         namespace: openshift-machine-api
       type: Opaque

1. Connect to the OpenShift Container Platform cluster.

2. Verify that the cluster does not have root credentials:

   $ oc get secrets -n kube-system aws-creds

   The output should look similar to:

   Error from server (NotFound): secrets "aws-creds" not found

3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

   Example command with the Image Registry Operator

   $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

   The output should show the role and web identity token that are used by the component and look similar to:

   Example output with the Image Registry Operator

   [default]
   role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
   web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token