Cloud-credential-operator-s3 not found in Openshift Container Platform 4.

Solution In Progress - Updated -

Environment

  • Openshift Container Platform 4 (OCP).
  • Cloud Credential Operator.

Issue

Cloud-credentials operator is degraded due to missing operator S3 creds.

2020-12-29T12:02:01.560687714Z time="2020-12-29T12:02:01Z" level=info msg="reconciling AWS S3 OIDC discovery endpoint" controller=oidcdiscoveryendpoint
2020-12-29T12:02:01.56923633Z time="2020-12-29T12:02:01Z" level=error msg="failed reconciling S3 resources" controller=oidcdiscoveryendpoint error="secrets \"cloud-credential-operator-s3-creds\" not found"

Resolution

The following permissions need to be recreated:

        s3:CreateBucket
        s3:PutBucketTagging
        s3:PutObject
        s3:PutObjectAcl

It can be done with the following yaml that needs to be created manually:

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: cloud-credential-operator-s3
  namespace: openshift-cloud-credential-operator
  annotations:
    exclude.release.openshift.io/internal-openshift-hosted: "true"
spec:
  secretRef:
    name: cloud-credential-operator-s3-creds
    namespace: openshift-cloud-credential-operator
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - s3:CreateBucket
      - s3:PutBucketTagging
      - s3:PutObject
      - s3:PutObjectAcl
      resource: "*"

Root Cause

The S3 CredentialsRequest object was not present in OCP 4.5. The operator in OCP 4.5 explicitly blocks the upgrade if the new secret doesn't exist as per following code.
The following pull request will potentially fix the behaviour in OCP 4.6.z:Link(pending to be checked)

Diagnostic Steps

Cloud Credential Operator pod logs throw a message warning:

msg="failed reconciling S3 resources" controller=oidcdiscoveryendpoint error="secrets \"cloud-credential-operator-s3-creds\" not found"

The creds are not found:

$ oc get credentialsrequest -n openshift-cloud-credential-operator cloud-credential-operator-s3
Error from server (NotFound): credentialsrequests.cloudcredential.openshift.io "cloud-credential-operator-s3" not found

The following secret should be available and linked on openshift-cloud-credential-operator namespace:

$ oc get secret cloud-credential-operator-s3-creds -n openshift-cloud-credential-operator -o json | jq -r .data
{
  "aws_access_key_id": "QUtJQVVxxxxxxx",
  "aws_secret_access_key": "ZWlDamVIdxxxxxxxxx",
  "credentials": "W2RlZmF1bHRdCmF3c19hY2Nlc3Nfa2V5X2lkID0gQUtJQVVNUUFIQ0pPQkxxxx"
}

It is possible to get the full list of programmatic users in AWS with the following check where "xxxxx" is the cluster identification:

aws iam list-users | grep "UserName.*openshift-xxxxx"

It is possible to associate the user with the corresponding secret following the name convention and checking whether it has proper permissions or not. Otherwise, it is possible to check directly the access key for each user and compare it with the access key stored in each secret. This command will give specify the access key for a specific user:

aws iam list-access-keys --query AccessKeyMetadata[].AccessKeyId --output text --user-name openshift-xxxxx-yyyy-xxxx-yyyyy

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments