Cloud Credentials Insufficient to Satisfy CredentialsRequest on AWS RHOCP 4
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- OCP-4 Cluster on AWS Cloud Platform
Issue
-
When attempting to transition to a distinct IAM user from a previously shared IAM user to help facilitate future credential rotation, the cloud-credential cluster operator showed as degraded:
$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.5.24 True False False 307d cloud-credential 4.5.24 True True True 308d
-
Below are the logs for the
cloud-credential-operator
:time="2021-02-10T00:59:44Z" level=info msg="calculating metrics for all CredentialsRequests" controller=metrics time="2021-02-10T00:59:44Z" level=info msg="reconcile complete" controller=metrics elapsed=1.374192ms time="2021-02-10T00:59:51Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds time="2021-02-10T00:59:51Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="running sync" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="Loading infrastructure name: ocp4-int-dev-test-xxxxx" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=error msg="cloud credentials insufficient to satisfy credentials request" actuator=aws cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro time="2021-02-10T00:59:51Z" level=error msg="error syncing credentials: cloud credentials insufficient to satisfy credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds time="2021-02-10T00:59:51Z" level=error msg="errored with condition: InsufficientCloudCreds" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds time="2021-02-10T00:59:51Z" level=debug msg="updating credentials request status" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds time="2021-02-10T00:59:51Z" level=debug msg="status unchanged" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds time="2021-02-10T00:59:51Z" level=debug msg="syncing cluster operator status" controller=credreq_status time="2021-02-10T00:59:51Z" level=debug msg="4 cred requests" controller=credreq_status time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="1 of 4 credentials requests are failing to sync." reason=CredentialsFailing status=True type=Degraded time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message="3 of 4 credentials requests provisioned, 1 reporting errors." reason=Reconciling status=True type=Progressing time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Available time="2021-02-10T00:59:51Z" level=debug msg="set ClusterOperator condition" controller=credreq_status message= reason= status=True type=Upgradeable
-
Cloud Credential Operator unable to reconcile the credentialsRequests with the mint mode for the cluster components like ingress and machine API and throwing below warnings related to the permissions in the CCO logs:
2023-08-25T15:16:11.404668982Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:PutObject" controller=secretannotator 2023-08-25T15:16:11.404684433Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:DeleteObject" controller=secretannotator 2023-08-25T15:16:11.404698673Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:ListBucketMultipartUploads" controller=secretannotator 2023-08-25T15:16:11.404698673Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="s3:AbortMultipartUpload" controller=secretannotator 2023-08-25T15:16:11.404716013Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeImages" controller=secretannotator 2023-08-25T15:16:11.404729903Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeVpcs" controller=secretannotator 2023-08-25T15:16:11.404764594Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeSubnets" controller=secretannotator 2023-08-25T15:16:11.404764594Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeAvailabilityZones" controller=secretannotator 2023-08-25T15:16:11.404781234Z time="2023-08-25T15:16:11Z" level=warning msg="Action not allowed with tested creds" action="ec2:DescribeSecurityGroups" controller=secretannotator 2023-08-25T15:16:11.405381417Z time="2023-08-25T15:16:11Z" level=warning msg="Cloud creds unable to be used for either minting or passthrough" controller=secretannotator
Resolution
Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.
-
One of the things to check would be for any Organization wide policies in place. In this case there was a Service Control Policy (SCP) attached to the AWS account which uses a global condition key. This causes the policy simulator to return access denied for just about everything which leads the operator to believe it does not have sufficient privileges.
-
Removing the SCP solved the issue.
-
If needed restart the Cloud-Credential-Operator pod to pick up the latest changes with the AWS Account and AWS IAM user.
Root Cause
-
SCPs are applied to an entire AWS account. They limit permissions for every request made by a principal within the account. An IAM entity (user or role) can make a request that is affected by an SCP, a permissions boundary, and an identity-based policy. In this case, the request is allowed only if all three policy types allow it. The effective permissions are the intersection of all three policy types. An explicit deny in any of these policies overrides the allow.
-
AWS strongly recommends that you don't attach SCPs to the root of your organization without thoroughly testing the impact that the policy has on accounts. Instead, create an OU that you can move your accounts into one at a time, or at least in small numbers, to ensure that you don't inadvertently lock users out of key services. One way to determine whether a service is used by an account is to examine the service last accessed data in IAM. Another way is to use AWS CloudTrail to log service usage at the API level.
Diagnostic Steps
Steps taken to diagnose the issue:
-
Create a new user, create access keys for the user & update the aws-creds secret in the kube-system namespace with the access keys for this new user.
-
Then, to test that it is working, delete the secret cloud-credential-operator-iam-ro-creds in the openshift-cloud-credential-operator namespace. What should happen is that the cloud-credential-operator should detect this, create new keys on the
-cloud-credential-operator-iam-ro- user, and create the secret in OCP. -
Unfortunately, forgetting to attach the IAM policy to the new IAM user caused this to fail. Even after attaching the policy the situation didn't improve. Suspecting that maybe the policy wasn't right, restoring the credentials for the original user which had the AdministratorAccess policy didn't help either.
-
Also creating the cloud-credential-operator-iam-ro-creds secret manually with empty secret values (which should also trigger the operator to update it) didn't work, even deleting the cloud-credential-operator pod to make sure that it wasn't stuck using the new IAM user's secrets held to no avail.
-
Grabbed
CredentialsRequest
object to see if there's any errors being reported:$ oc get credentialsrequest -n openshift-cloud-credential-operator cloud-credential-operator-iam-ro -o yaml apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: annotations: exclude.release.openshift.io/internal-openshift-hosted: "true" creationTimestamp: "2020-12-03T14:24:53Z" finalizers: - cloudcredential.openshift.io/deprovision generation: 1 labels: controller-tools.k8s.io: "1.0" managedFields: - apiVersion: cloudcredential.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:exclude.release.openshift.io/internal-openshift-hosted: {} f:labels: .: {} f:controller-tools.k8s.io: {} f:spec: .: {} f:providerSpec: .: {} f:apiVersion: {} f:kind: {} f:statementEntries: {} f:secretRef: .: {} f:name: {} f:namespace: {} manager: cluster-version-operator operation: Update time: "2020-12-03T14:24:53Z" - apiVersion: cloudcredential.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"cloudcredential.openshift.io/deprovision": {} f:status: .: {} f:conditions: {} f:lastSyncGeneration: {} f:lastSyncTimestamp: {} f:providerStatus: .: {} f:apiVersion: {} f:kind: {} f:policy: {} f:user: {} f:provisioned: {} manager: cloud-credential-operator operation: Update time: "2021-02-10T00:52:57Z" name: cloud-credential-operator-iam-ro namespace: openshift-cloud-credential-operator resourceVersion: "2764xxxxx" selfLink: /apis/cloudcredential.openshift.io/v1/namespaces/openshift-cloud-credential-operator/credentialsrequests/cloud-credential-operator-iam-ro uid: e59626e8-6181-4181-86c7-8xxxxxxxxxx spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - action: - iam:GetUser - iam:GetUserPolicy - iam:ListAccessKeys effect: Allow resource: '*' secretRef: name: cloud-credential-operator-iam-ro-creds namespace: openshift-cloud-credential-operator status: conditions: - lastProbeTime: "2021-02-10T00:52:57Z" lastTransitionTime: "2021-02-10T00:52:57Z" message: cloud creds are insufficient to satisfy CredentialsRequest reason: CloudCredsInsufficient status: "True" type: InsufficientCloudCreds - lastProbeTime: "2021-02-10T00:52:57Z" lastTransitionTime: "2021-02-10T00:52:57Z" message: successfully granted credentials request reason: CredentialsProvisionSuccess status: "False" type: CredentialsProvisionFailure lastSyncGeneration: 1 lastSyncTimestamp: "2021-02-09T23:20:09Z" providerStatus: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderStatus policy: ocp4-int-dev-test-mj-cloud-credential-operator-iam-ro-xxxxx-policy user: ocp4-int-dev-test-mj-cloud-credential-operator-iam-ro-xxxxx provisioned: false
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments