oc login from remote failed towards Private Link ROSA cluster

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4

Issue

  • When using oc login from remote client to Private Link ROSA cluster get below two kinds of error from debug log
# oc login https://api.xxxxxx.openshiftapps.com:6443 --username cluster-admin --password xxxxxxxxxxxxxxxxxxxx -v=10
I0324 18:07:03.905972   25092 round_trippers.go:466] curl -v -XHEAD  'https://api.xxxxxx.openshiftapps.com:6443/'
I0324 18:07:03.906370   25092 round_trippers.go:495] HTTP Trace: DNS Lookup for api.xxxxxx.openshiftapps.com resolved to [{xx.xxx.xxx.xx }]
I0324 18:07:03.907822   25092 round_trippers.go:510] HTTP Trace: Dial to tcp:xx.xxx.xxx.xx:6443 succeed
.........
I0324 18:07:03.990167   25092 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
I0324 18:07:03.990392   25092 helpers.go:264] Connection error: Head https://oauth-xxxxxx.openshiftapps.com/: stopped after 10 redirects
Unable to connect to the server: stopped after 10 redirects

oc login https://api.xxxxxx.openshiftapps.com:6443 --username cluster-admin --password xxxxxxxxxxxxxxxxxxxx --loglevel=10
I0308 22:25:11.926798   13903 round_trippers.go:466] curl -v -XHEAD  'https://api.xxxxxx.openshiftapps.com:6443/'
I0308 22:25:11.928765   13903 round_trippers.go:495] HTTP Trace: DNS Lookup for api.xxxxxx.openshiftapps.com resolved to [{xx.xxx.xxx.xx } ]
I0308 22:25:11.932697   13903 round_trippers.go:510] HTTP Trace: Dial to tcp:xx.xxx.xxx.xx:6443 succeed
.........
I0308 22:25:11.979349   13903 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
I0308 22:25:11.979761   13903 helpers.go:264] Connection error: Head https://oauth-xxxxxx.openshiftapps.com: Forbidden
Unable to connect to the server: Forbidden

Resolution

  • Change network setting so that remote oc client can connect to oauth domain and using curl -kvs https://oauth-xxxxxx.openshiftapps.com/healthz to check if the network connection is successful.

  • Set up noproxy towards oauth-xxxxxx.openshiftapps.com can help to divide if it is proxy related setting issue.

Root Cause

  • There are network connection issue exist from remote oc client to Private Link ROSA cluster oauth domain

Diagnostic Steps

  • Using curl to check connection from oc client to oauth health check URL get error
# curl -kvs https://oauth-xxxxxx.openshiftapps.com/healthz
* About to connect() to oauth-xxxxxx.openshiftapps.com port 443 (#0)
*   Trying xx.xxx.x.xxx...
* Connected to oauth-xxxxxx.openshiftapps.com (xx.xxx.x.xxx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -8054 (SEC_ERROR_REUSED_ISSUER_AND_SERIAL)
* You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert.
* Closing connection 0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments