TLS connection fails connecting to services hosted on ARO domains

Solution Verified - Updated -

Environment

  • Azure Red Hat Openshift (ARO)
    • v4.x

Issue

  • Connections to cluster API service fails with TLS errors when using an intermediate proxy / firewall with outdated root CA trust store

  • Connections to routes provided by default ingress routers fail with TLS errors when using an intermediate proxy / firewall with outdated root CA trust store

  • TLS connection errors reported by client include certificate not trusted

Answer: ERROR26/May/2025:01:23:27.134 no trusted certificates on chain for this context:
{ subject: /C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.apps.mycluster.myregion.aroapp.io, thumbprint: }
{ subject: /C=US/O=Microsoft Corporation/CN=Microsoft Azure RSA TLS Issuing CA 08, thumbprint: }
{ subject: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root G2, thumbprint: }
ERROR26/May/2025:01:23:27.134[SSL alert write 0x22a, 0x4]: bad certificate [fatal].
ERROR26/May/2025:01:23:27.134[SSL_connect, 0x4]: error - certificate not trusted.
ERROR26/May/2025:01:23:27.134transient failure connecting to remote: SSL protocol error
error:0A000086:SSL routines::certificate verify failed, source location: sslctx.cpp:485 (unknown function)
error:0A000086:SSL routines::certificate verify failed, source location: ssl/statem/statem_clnt.c:1890

Resolution

  • Add the new root certificate (DigiCert Global Root G2) to the intermediate proxy / firewall as trusted. This will restore trust in the full certificate chain presented by the ingress router.

  • Use an additional / separate (non-default) ingress controller, supplying Customer-managed certificates for Customer workload ingress connections.

Root Cause

In 2021, Microsoft updated Azure services to use TLS certificates from Root Certificate Authorities (CA) where use of the SHA-1 algorithm had ended, migrating from "DigiCert Global Root CA" to "DigiCert Global Root G2" [ 1 ].

As part of the managed service, certificate rotation of the Azure Red Hat OpenShift platform is performed periodically as standard platform maintenance [ 2 ].

In environments where an intermediate proxy / firewall has been implemented to perform certificate verification between the client and Azure Red Hat OpenShift cluster, it is necessary to ensure that the intermediate CA trust store includes the currently supported certificates.

This change is performed automatically as part of Azure Red Hat OpenShift platform certificate rotation (standard behavior for managed clusters).

Please note that Customers who have not implemented an intermediate proxy / firewall with root CA certificate validation checking will not experience this issue.

Note: Connections will fail at the TLS handshake (as shown in Issue section) if the newer root CA is not trusted by the Customer-managed intermediate proxy / firewall.

Disclaimer: Above link contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments