Red Hat OpenShift Upgrade Blocked in Hosted Control Planes Due to Azure OIDC Certificate Validation Problems

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.16
  • Hosted control planes

Issue

  • In Hosted control planes, when upgrading OpenShift from 4.15 to 4.16, the upgrade process may get stuck, with no visible errors in the standard output or web console. The only indication of failure can be found in the hosted-control-plane-operator logs, showing a certificate validation error:

    x509: certificate signed by unknown authority

  • This issue is caused by OpenShift 4.16 enforcing stricter certificate validation, requiring the Azure CA certificate to be explicitly trusted, even though it is a globally trusted CA.

Resolution

  • To resolve the Azure CA certificate validation issue in OpenShift 4.16, manually add the Azure CA to the OAuth configuration.

  • First, use the following command to fetch the Azure CA certificate chain:

    $ openssl s_client -connect login.microsoftonline.com:443 -showcerts
    • This command will display the certificate chain, including the intermediate and root certificates. Save the root certificate to a file (e.g., azure-ca.crt).

  • Next, create a ConfigMap to store the Azure CA certificate in the appropriate OpenShift namespace:

    $ oc create configmap azure-ca --from-file=ca.crt=azure-ca.crt -n <namespace>

  • Now, update the OAuth configuration by adding the ca field to the identityProviders.openID section. This field should reference the ConfigMap.

    oauth:
  identityProviders:
  - name: azure
    openID:
      clientID: <client-id>
      clientSecret:
        name: <secret-name>
      issuer: https://login.microsoftonline.com/<tenant-id>/v2.0
      ca: azure-ca
    type: OpenID

  • Once the configuration has been updated, restart the OAuth pods to apply the changes.

    $ oc delete pod -n openshift-authentication -l app=oauth-openshift

  • After the pods have restarted, retry the upgrade process. The issue with certificate validation should be resolved, allowing the upgrade to proceed.

Root Cause

  • OpenShift 4.16 introduced stricter certificate validation policies for external OIDC providers.

  • Even though Microsoft’s Azure CA is globally trusted and valid until 2030, the new policy requires an explicit trust configuration for the Azure CA, which was not needed in OpenShift 4.15. This results in the upgrade process being blocked without proper indication in standard logs or the UI.

Diagnostic Steps

  • Check control-plane-operator logs for the following error:

    $ oc logs -n <namespace> $(oc get pods -n <namespace> -l app=control-plane-operator -o name)

  • Look for:

    x509: certificate signed by unknown authority

  • Check OAuth configuration to see if the ca field is missing from the identityProviders section.

  • Verify the certificate chain by running:

    $ openssl s_client -connect login.microsoftonline.com:443 -showcerts

Note:

  • There is no immediate urgency to update the Azure CA manually, as the root CA is valid until 2030.

  • A product improvement may be considered to handle globally trusted CAs dynamically in future OpenShift versions.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments