Unable to access Key Vault due to authentication error

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4

Issue

  • When an app running in one namespace is able to authenticate, an app running in the other namespace is unable to
    access Key Vault due to an authentication error.

Background

  • Created a service principal to use Key Vault from the pod of ARO and assigned permissions to Key Vault. Multiple applications access the same Key Vault in the following steps.

    1. Connection from the application to Key Vault is made by placing the
      client ID and secret in the Secret of ARO.
    2. When the application starts, read the key from the environment
      variable and issue a credential in the SDK.
    3. Access Key Vault with that credential.

  • The apps are launched in multiple namespaces, and each namespace contains a Secret that stores the client ID and secret of the same service principal. When access checks were performed using the above configuration, the following error occurred when the app running in one namespace was authenticated, but the app running in the other namespace could not access Key Vault due to an authentication error.

AuthenticationError: EnvironmentCredential authentication failed. To troubleshoot, visit https://aka.ms/azsdk/js/identity/environmentcredential/troubleshoot. Status code: 400
More details:
endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: ClientConfigurationError: untrusted_authority: The provided authority is not a trusted authority. Please include this authority in the knownAuthorities config parameter.
{
  statusCode: 400,
  errorResponse: {
    error: 'EnvironmentCredential authentication failed. To troubleshoot, visit https://aka.ms/azsdk/js/identity/environmentcredential/troubleshoot.',
    errorDescription: 'endpoints_resolution_error: Error: could not resolve endpoints. Please check network and try again. Detail: ClientConfigurationError: untrusted_authority: The provided authority is not a trusted authority. Please include this authority in the knownAuthorities config parameter.',
    correlationId: undefined,
    errorCodes: undefined,
    timestamp: undefined,
    traceId: undefined
  }
}

Resolution

Disclaimer: link contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • In the troubleshoot link of the error message, there is a similar "endpoints_resolution_error". It mentioned as mitigation, "Ensure the specified tenantId is correct for your application registration. For multi-tenant apps, ensure the application has been added to the desired tenant by a tenant admin. To add a new application in the desired tenant, follow the instructions here.".
    • Here is another way to use Azure Key Vault Provider for Secrets Store CSI Driver on ARO like below. To divide the issue, we can also try referencing the below sample to set up the same Key Vault in a different namespace and see if the problem can be solved.
cat <<EOF | kubectl apply -f -
 apiVersion: secrets-store.csi.x-k8s.io/v1
 kind: SecretProviderClass
 metadata:
   name: azure-kvname
   namespace: my-application
 spec:
   provider: azure
   parameters:
     usePodIdentity: "false"
     useVMManagedIdentity: "false"
     userAssignedIdentityID: ""
     keyvaultName: "${KEYVAULT_NAME}"
     objects: |
       array:
         - |
           objectName: secret1
           objectType: secret
           objectVersion: ""
     tenantId: "${AZ_TENANT_ID}"
 EOF

cat <<EOF | kubectl apply -f -
 kind: Pod
 apiVersion: v1
 metadata:
   name: busybox-secrets-store-inline
   namespace: my-application
 spec:
   containers:
   - name: busybox
     image: k8s.gcr.io/e2e-test-images/busybox:1.29
     command:
       - "/bin/sleep"
       - "10000"
     volumeMounts:
     - name: secrets-store-inline
       mountPath: "/mnt/secrets-store"
       readOnly: true
   volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "azure-kvname"
         nodePublishSecretRef:
           name: secrets-store-creds
 EOF

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments