Rotate Azure service principal credentials for ARO

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4

Issue

  • ARO Service Principal expired and needs to be rotated.
  • ARO Service Principal is about to expire.
  • Error messages like the following one are shown in ARO cluster:

    Failed to refresh the Token for request to https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/arocluster_cluster/providers/Microsoft.Compute/disks/arocluster-xxxxx-dynamic-pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx?api-version=2019-11-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {'error':'invalid_client','error_description':'AADSTS7000222: The provided client secret keys for app 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds
    

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

It is possible to use the az aro update command with the flag --refresh-credentials to rotate the service principal credentials. Refer to Rotate service principal credentials for your Azure Red Hat OpenShift (ARO) Cluster for additional information.

Refer also to Azure AD recommendation for renewing expiring service principal credentials in the Azure documentation.

NOTE: The secret must not be edited manually in the Azure Portal. Running the command as provided in the documentation is the correct approach in order to have the required reconciliations in the backend.

Root Cause

The Azure service principal is an Azure resource and it is not managed or monitored by the Red Hat SRE Team due to permission restrictions. Is customer responsibility to monitor and rotate it when needed.

The automated service principal credential rotation with the az aro update command will check if the service principal exists and rotate or create a new service principal.

Diagnostic Steps

  • To check the expiration date of service principal credentials run the following:
# Service principal expiry in UTC format
$ az ad app credential list --id $(az aro show --name [MyManagedCluster] --resource-group [MyResourceGroup] --query servicePrincipalProfile.clientId -o tsv) --query "[].endDateTime" -o tsv
  • To check if the secret matches on OpenShift endpoint.
echo ; oc get secret azure-credentials -n kube-system --template='{{range $k, $v := .data }} {{ $k }} {{ " : " }} {{ $v | base64decode }} {{ "\n" }} {{ end }}' ; echo

echo ; oc get secret azure-cloud-provider -n kube-system --template='{{range $k, $v := .data }} {{ $k }} {{ " : " }} {{ $v | base64decode }} {{ "\n" }} {{ end }}' ; echo

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments