OADP on ROSA fails with "NoCredentialProviders: no valid providers in chain. Deprecated."

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA) 4.11
  • AWS Security Token Service (STS)
  • OpenShift APIs for Data Protection (OADP)

Issue

  • When trying to use the AWS Security Token Service (STS) with Red Hat OpenShift Service on AWS (ROSA) and the OpenShift APIs for Data Protection (OADP), this fails due to the BackupStorageLocation showing the following error:

    status:
  lastValidationTime: "2023-01-18T15:19:40Z"
  message: "BackupStorageLocation \"example-1\" is unavailable: rpc error: code =
    Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor
    verbose messaging see aws.Config.CredentialsChainVerboseErrors"
  phase: Unavailable

  • When using S3 bucket encryption, the BackupStorageLocation also shows the following error message when trying to configure STS for OADP on ROSA:

    Last Validation Time:  2023-01-24T21:57:42Z
Message:               BackupStorageLocation "backup-dpa-1" is unavailable: rpc error: code = Unknown desc = WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
      status code: 403, request id: f6xxxa01-b94e-4950-ac94-xxxxxab8f26

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • Follow the (unsupported) documentation to set up OADP on a ROSA STS cluster: https://mobb.ninja/docs/misc/oadp/rosa-sts

  • Make sure to use the kmsKeyId in the DataProtectionApplication is set:

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: example-dpa
  namespace: example-adp
spec:
  backupLocations:
  - bucket:
      cloudStorageRef:
        name: example-rosa-oadp
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      config:
        kmsKeyId: kms-key-id
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - aws

  • Also, verify that the trust policy used specifies the correct AWS region (eu-central-1 in the below example):

    {
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::1234567890:oidc-provider/rh-oidc.s3.eu-central-1.amazonaws.com/2EXAMPLEtofEXAMPLE2ohkEXAMPLE"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
            "StringEquals": {
                "rh-oidc.s3.eu-central-1.amazonaws.com/2EXAMPLEtofEXAMPLE2ohkEXAMPLE:sub": [
                    "system:serviceaccount:example-adp:openshift-adp-controller-manager",
                    "system:serviceaccount:example-adp:velero"
                ]
            }
        }
    }
]
}

Root Cause

  • The KMS encryption key and trust policy are not set up correctly.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments