Cloud Credentials Insufficient to Satisfy CredentialsRequest on AWS RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • OCP-4 Cluster on AWS Cloud Platform

Issue

  • When attempting to transition to a distinct IAM user from a previously shared IAM user to help facilitate future credential rotation, the cloud-credential cluster operator showed as degraded:

    $ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.24    True        False         False      307d
cloud-credential                           4.5.24    True        True          True       308d

  • Below are the logs for the cloud-credential-operator :

    time="2021-02-10T00:59:44Z" level=info msg="calculating metrics for all CredentialsRequests" controller=metrics
time="2021-02-10T00:59:44Z" level=info msg="reconcile complete" controller=metrics elapsed=1.374192ms
time="2021-02-10T00:59:51Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2021-02-10T00:59:51Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="running sync" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="Loading infrastructure name: ocp4-int-dev-test-xxxxx" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=error msg="cloud credentials insufficient to satisfy credentials request" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro
time="2021-02-10T00:59:51Z" level=error msg="error syncing credentials: cloud credentials insufficient to satisfy credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2021-02-10T00:59:51Z" level=error msg="errored with condition: InsufficientCloudCreds" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2021-02-10T00:59:51Z" level=debug msg="updating credentials request status" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2021-02-10T00:59:51Z" level=debug msg="status unchanged" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2021-02-10T00:59:51Z" level=debug msg="syncing cluster operator status" controller=credreq_status
time="2021-02-10T00:59:51Z" level=debug msg="4 cred requests" controller=credreq_status
time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="1 of 4 credentials requests are failing to sync." reason=CredentialsFailing status=True type=Degraded
time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="3 of 4 credentials requests provisioned, 1 reporting errors." reason=Reconciling status=True type=Progressing
time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Available
time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Upgradeable

  • Cloud Credential Operator unable to reconcile the credentialsRequests with the mint mode for the cluster components like ingress and machine API and throwing below warnings related to the permissions in the CCO logs:

    2023-08-25T15:16:11.404668982Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:PutObject" controller=secretannotator
2023-08-25T15:16:11.404684433Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:DeleteObject" controller=secretannotator
2023-08-25T15:16:11.404698673Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:ListBucketMultipartUploads" controller=secretannotator
2023-08-25T15:16:11.404698673Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:AbortMultipartUpload" controller=secretannotator
2023-08-25T15:16:11.404716013Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeImages" controller=secretannotator
2023-08-25T15:16:11.404729903Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeVpcs" controller=secretannotator
2023-08-25T15:16:11.404764594Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeSubnets" controller=secretannotator
2023-08-25T15:16:11.404764594Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeAvailabilityZones" controller=secretannotator
2023-08-25T15:16:11.404781234Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeSecurityGroups" controller=secretannotator
2023-08-25T15:16:11.405381417Z time="2023-08-25T15:16:11Z" level=warning msg="Cloud creds unable to be used for either minting or passthrough" controller=secretannotator

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • One of the things to check would be for any Organization wide policies in place. In this case there was a Service Control Policy (SCP) attached to the AWS account which uses a global condition key. This causes the policy simulator to return access denied for just about everything which leads the operator to believe it does not have sufficient privileges.

  • Removing the SCP solved the issue.

  • If needed restart the Cloud-Credential-Operator pod to pick up the latest changes with the AWS Account and AWS IAM user.

Root Cause

  • Organizations SCPs :

    SCPs are applied to an entire AWS account. They limit permissions for every request made by a principal within the account. An IAM entity (user or role) can make a request that is affected by an SCP, a permissions boundary, and an identity-based policy. In this case, the request is allowed only if all three policy types allow it. The effective permissions are the intersection of all three policy types. An explicit deny in any of these policies overrides the allow.

  • AWS strongly recommends that you don't attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services. One way to determine whether a service is used by an account is to examine the service last accessed data in IAM. Another way is to use AWS CloudTrail to log service usage at the API level.

Diagnostic Steps

Steps taken to diagnose the issue:

  • Create a new user, create access keys for the user & update the aws-creds secret in the kube-system namespace with the access keys for this new user.

  • Then, to test that it is working, delete the secret cloud-credential-operator-iam-ro-creds in the openshift-cloud-credential-operator namespace. What should happen is that the cloud-credential-operator should detect this, create new keys on the -cloud-credential-operator-iam-ro- user, and create the secret in OCP.

  • Unfortunately, forgetting to attach the IAM policy to the new IAM user caused this to fail. Even after attaching the policy the situation didn't improve. Suspecting that maybe the policy wasn't right, restoring the credentials for the original user which had the AdministratorAccess policy didn't help either.

  • Also creating the cloud-credential-operator-iam-ro-creds secret manually with empty secret values (which should also trigger the operator to update it) didn't work, even deleting the cloud-credential-operator pod to make sure that it wasn't stuck using the new IAM user's secrets held to no avail.

  • Grabbed CredentialsRequest object to see if there's any errors being reported:

    $ oc get credentialsrequest -n openshift-cloud-credential-operator cloud-credential-operator-iam-ro -o yaml
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
annotations:
  exclude.release.openshift.io/internal-openshift-hosted: "true"
creationTimestamp: "2020-12-03T14:24:53Z"
finalizers:
- cloudcredential.openshift.io/deprovision
generation: 1
labels:
  controller-tools.k8s.io: "1.0"
managedFields:
- apiVersion: cloudcredential.openshift.io/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:metadata:
      f:annotations:
        .: {}
        f:exclude.release.openshift.io/internal-openshift-hosted: {}
      f:labels:
        .: {}
        f:controller-tools.k8s.io: {}
    f:spec:
      .: {}
      f:providerSpec:
        .: {}
        f:apiVersion: {}
        f:kind: {}
        f:statementEntries: {}
      f:secretRef:
        .: {}
        f:name: {}
        f:namespace: {}
  manager: cluster-version-operator
  operation: Update
  time: "2020-12-03T14:24:53Z"
- apiVersion: cloudcredential.openshift.io/v1
  fieldsType: FieldsV1
  fieldsV1:
    f:metadata:
      f:finalizers:
        .: {}
        v:"cloudcredential.openshift.io/deprovision": {}
    f:status:
      .: {}
      f:conditions: {}
      f:lastSyncGeneration: {}
      f:lastSyncTimestamp: {}
      f:providerStatus:
        .: {}
        f:apiVersion: {}
        f:kind: {}
        f:policy: {}
        f:user: {}
      f:provisioned: {}
  manager: cloud-credential-operator
  operation: Update
  time: "2021-02-10T00:52:57Z"
name: cloud-credential-operator-iam-ro
namespace: openshift-cloud-credential-operator
resourceVersion: "2764xxxxx"
selfLink: /apis/cloudcredential.openshift.io/v1/namespaces/openshift-cloud-credential-operator/credentialsrequests/cloud-credential-operator-iam-ro
uid: e59626e8-6181-4181-86c7-8xxxxxxxxxx
spec:
providerSpec:
  apiVersion: cloudcredential.openshift.io/v1
  kind: AWSProviderSpec
  statementEntries:
  - action:
    - iam:GetUser
    - iam:GetUserPolicy
    - iam:ListAccessKeys
    effect: Allow
    resource: '*'
secretRef:
  name: cloud-credential-operator-iam-ro-creds
  namespace: openshift-cloud-credential-operator
status:
conditions:
- lastProbeTime: "2021-02-10T00:52:57Z"
  lastTransitionTime: "2021-02-10T00:52:57Z"
  message: cloud creds are insufficient to satisfy CredentialsRequest
  reason: CloudCredsInsufficient
  status: "True"
  type: InsufficientCloudCreds
- lastProbeTime: "2021-02-10T00:52:57Z"
  lastTransitionTime: "2021-02-10T00:52:57Z"
  message: successfully granted credentials request
  reason: CredentialsProvisionSuccess
  status: "False"
  type: CredentialsProvisionFailure
lastSyncGeneration: 1
lastSyncTimestamp: "2021-02-09T23:20:09Z"
providerStatus:
  apiVersion: cloudcredential.openshift.io/v1
  kind: AWSProviderStatus
  policy: ocp4-int-dev-test-mj-cloud-credential-operator-iam-ro-xxxxx-policy
  user: ocp4-int-dev-test-mj-cloud-credential-operator-iam-ro-xxxxx
provisioned: false

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments