rosa commands fails on Windows - can't send request: x509: certificate signed by unknown authority

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
  • Windows platform
  • rosa CLI for Windows < v1.1.5

Issue

  • Running the ROSA install from a Windows platform fails with a similar an error similar to this:

    ERR: Failed to retrieve versions: Get "https://api.openshift.com/api/clusters_mgmt/v1/versions?order=default+desc%2C+id+desc&page=1&search=enabled+%3D+%27true%27+AND+rosa_enabled+%3D+%27true%27+AND+channel_group+%3D+%27stable%27&size=100 ": can't send request: x509: certificate signed by unknown authority

  • Running rosa commands fails in Windows platform with errors similar to:

    ERR: Failed to retrieve versions: Get "https://api.openshift.com/api/clusters_mgmt/v1/clusters?xxxxxxxxxxxxxxxxxxx: can't send request: x509: certificate signed by unknown authority

Resolution

This is a known bug of the rosa CLI for the Windows platform. Version 1.1.5 of the rosa CLI was already released to mitigate this issue .

Note: In some cases, the fix included in rosa version 1.1.5 is still failing with the same error. Please, use the workaround shown below.

To check the version:

rosa.exe version
1.1.3

To access the latest version for your platform, download the rosa CLI from the OpenShift Downloads page.

Workaround

It's possible to use the rosa CLI on another platform (Linux or MacOS) as these platforms are not impacted by this issue.

Only as a workaround, it's also possible to use the --insecure flag when running a rosa CLI version with the issue on Windows:

rosa.exe login --insecure

Root Cause

The rosa CLI isn't using the system root CA / trust store on Windows.

Instead, it uses embedded certificates in the ocm go package (ocm sdk) to validate the TLS connections for api.openshift.com and Red Hat's Customer Portal (SSO)

Diagnostic Steps

The rosa CLI requires the the ocm sdk which requires the net/http which requires the crypto/x509 default go packages.

Due to an issue with crypto/x509 being unable to access the Windows root CA bundle / trust store, it was left in the hands of developers using the crypto/x509 package to embed CA certificates into their packages.

From the x509 source:

func loadSystemRoots() (*CertPool, error) {
    // TODO: restore this functionality on Windows. We tried to do
    // it in Go 1.8 but had to revert it. See Issue 18609.
    // Returning (nil, nil) was the old behavior, prior to CL 30578.
    // The if statement here avoids vet complaining about
    // unreachable code below.
    if true {
        return nil, nil
    }
...
}

This issue 18609 effectively forces the loadSystemRoots() function to load a nil object on every call. Meaning that no root CAs can be picked up from the Windows OS.

This is why the ocm package / sdk embeds the SSO and Lets Encrypt Root CA's. There are more detail in the pull request #333 to workaround the x509 issue.

However the root CA changed for Lets Encrypt (September 30 14:01:15 2021 GMT), so a second pull request#484 was created to embed the 2nd CA into the ocm go package. We are currently waiting for this to be packaged into a new rosa CLI version.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments