Secret "ebs-cloud-credentials" Not Found on ROSA

Solution Verified - Updated -

Environment

  • Red Hat OpenShift on AWS (ROSA)
    • 4

Issue

  • Persistent volumes are not being provisioned with the event: MountVolume.SetUp failed for volume "aws-credentials" : secret "ebs-cloud-credentials" not found
  • The aws-ebs-csi-controllers are stuck in a ContainerCreating status.
  • Logs indicate that the "ebs-cloud-credentials" secret is missing.

Resolution

  1. Retrieve the Role ARN:
    Use the rosa CLI tool to fetch the Role ARN for the EBS CSI driver:

    ROLE_ARN=$(rosa describe cluster -c YOUR_CLUSTER_ID -o json | jq -r '.aws.sts.operator_iam_roles[] | select(.name == "ebs-cloud-credentials") | .role_arn')

  2. Prepare the Credentials:
    Create the unencoded credentials content:

    CREDENTIALS_TXT=$(cat <<EOF
[default]
role_arn = ${ROLE_ARN}
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
sts_regional_endpoints = regional
EOF
)

  3. Base64 Encode the Credentials:
    Ensure consistent encoding across platforms:

    OS=$(uname)
BASE64_OPT=""
if [ "${OS}" == "Linux" ]; then
   BASE64_OPT="-w0"
fi
ENCODED_CREDENTIALS=$(printf "%s" "${CREDENTIALS_TXT}" | base64 ${BASE64_OPT})
ENCODED_ROLE_ARN=$(printf "%s" "${ROLE_ARN}" | base64 ${BASE64_OPT})

Note: The -w0 option is specific to the GNU version of base64 (AKA Linux). On macOS, the equivalent option is -b, but it's not needed because base64 on macOS does not wrap lines by default.

  1. Recreate the Secret Using the Encoded Values:
    Create a YAML manifest for the secret:

    cat <<EOF > ebs-cloud-credentials-secret.yaml
apiVersion: v1
kind: Secret
metadata:
 name: ebs-cloud-credentials
 namespace: openshift-cluster-csi-drivers
type: Opaque
data:
 credentials: ${ENCODED_CREDENTIALS}
 role_arn: ${ENCODED_ROLE_ARN}
EOF

    Apply the secret using the oc CLI tool:

    oc apply -f ebs-cloud-credentials-secret.yaml

Root Cause

The ebs-cloud-credentials secret, which contains the AWS Role ARN needed by the EBS CSI driver, was deleted. This secret is not automatically reconciled and is typically only created during cluster installation.

This was usually caused by the namespace openshift-cluster-csi-drivers has been accidentally deleted.

Diagnostic Steps

  1. Check if the Namespace was Recently Deleted and Recreated:
    Review the timestamp of openshift-cluster-csi-drivers namespace:

    oc get namespace openshift-cluster-csi-drivers

  2. Verify if the Secret is Missing:
    Check for the presence of the ebs-cloud-credentials Secret:

    oc get secret ebs-cloud-credentials -n openshift-cluster-csi-drivers

    If the Secret is missing, you will receive a "not found" error.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments