authentication operator degraded
I just installed a bare metal three node cluster (version 4.12.5) on three BL460 HP blades, following the instructions at Installing a user-provisioned bare metal cluster on a restricted network, but the authentication operator shows degraded.
$ oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.12.5 False False True 3h51m OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.app
s.csvocp.csv.uschi.nsn-rdnet.net/healthz": x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
baremetal 4.12.5 True False False 3h50m
cloud-controller-manager 4.12.5 True False False 4h
cloud-credential 4.12.5 True False False 4h57m
cluster-autoscaler 4.12.5 True False False 3h51m
config-operator 4.12.5 True False False 3h51m
console 4.12.5 False True False 3h41m DeploymentAvailable: 0 replicas available for console deployment...
control-plane-machine-set 4.12.5 True False False 3h50m
csi-snapshot-controller 4.12.5 True False False 3h51m
dns 4.12.5 True False False 3h50m
etcd 4.12.5 True False False 3h49m
image-registry 4.12.5 True False False 3h42m
ingress 4.12.5 True False True 3h43m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more
other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
insights 4.12.5 False False True 124m Unable to report: unable to build request to connect to Insights server: Post "https://
console.redhat.com/api/ingress/v1/upload": dial tcp: lookup console.redhat.com on 172.30.0.10:53: no such host
kube-apiserver 4.12.5 True False False 3h43m
kube-controller-manager 4.12.5 True False False 3h48m
kube-scheduler 4.12.5 True False False 3h43m
kube-storage-version-migrator 4.12.5 True False False 3h51m
machine-api 4.12.5 True False False 3h50m
machine-approver 4.12.5 True False False 3h50m
machine-config 4.12.5 True False False 3h49m
marketplace 4.12.5 True False False 3h50m
monitoring 4.12.5 True False False 3h39m
network 4.12.5 True False False 3h52m
node-tuning 4.12.5 True False False 3h51m
openshift-apiserver 4.12.5 True False False 3h37m
openshift-controller-manager 4.12.5 True False False 3h42m
openshift-samples 4.12.5 True False False 3h36m
operator-lifecycle-manager 4.12.5 True False False 3h51m
operator-lifecycle-manager-catalog 4.12.5 True False False 3h51m
operator-lifecycle-manager-packageserver 4.12.5 True False False 3h43m
service-ca 4.12.5 True False False 3h51m
storage 4.12.5 True False False 3h51m
As a consequence, as seen in the snipped above too, the console pods cannot start either, and I can see messages like this in their logs.
$ oc logs -n openshift-console console-59857c965f-v97vf | tail -n 3
E0310 20:20:10.649939 1 auth.go:232] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net/oauth/token failed: Head "https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net": x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
E0310 20:20:20.660907 1 auth.go:232] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net/oauth/token failed: Head "https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net": x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
E0310 20:20:30.669558 1 auth.go:232] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net/oauth/token failed: Head "https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net": x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
I tried replacing the ingress certificates as described at Replacing the default ingress certificate, but it didn't help.
I believe the problem I am having trying to log in is related to this, too.
$ oc login -u kubeadmin -p pySNL-HUIJL-GJIb7-TD5zg https://api.csvocp.csv.uschi.nsn-rdnet.net:6443 --loglevel=9
I0310 14:25:22.352362 141180 loader.go:374] Config loaded from file: csvocp/auth/kubeconfig
I0310 14:25:22.353031 141180 round_trippers.go:466] curl -v -XHEAD 'https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/'
I0310 14:25:22.354514 141180 round_trippers.go:495] HTTP Trace: DNS Lookup for api.csvocp.csv.uschi.nsn-rdnet.net resolved to [{10.4.185.75 }]
I0310 14:25:22.354825 141180 round_trippers.go:510] HTTP Trace: Dial to tcp:10.4.185.75:6443 succeed
I0310 14:25:22.360841 141180 round_trippers.go:553] HEAD https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/ 403 Forbidden in 7 milliseconds
I0310 14:25:22.360880 141180 round_trippers.go:570] HTTP Statistics: DNSLookup 1 ms Dial 0 ms TLSHandshake 4 ms ServerProcessing 0 ms Duration 7 ms
I0310 14:25:22.360899 141180 round_trippers.go:577] Response Headers:
I0310 14:25:22.360921 141180 round_trippers.go:580] Date: Fri, 10 Mar 2023 20:25:30 GMT
I0310 14:25:22.360940 141180 round_trippers.go:580] Cache-Control: no-cache, private
I0310 14:25:22.360958 141180 round_trippers.go:580] Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I0310 14:25:22.360977 141180 round_trippers.go:580] X-Content-Type-Options: nosniff
I0310 14:25:22.360997 141180 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 3d29411f-1f04-4fd6-a3ff-7aa305d1d239
I0310 14:25:22.361015 141180 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: dc5e6c2c-3bd5-41b4-9a4c-5acf0ed9a449
I0310 14:25:22.361034 141180 round_trippers.go:580] Audit-Id: d60c46fe-31d0-4cf5-9e92-7df793b2a80d
I0310 14:25:22.361052 141180 round_trippers.go:580] Content-Type: application/json
I0310 14:25:22.361070 141180 round_trippers.go:580] Content-Length: 186
I0310 14:25:22.361125 141180 request_token.go:93] GSSAPI Enabled
I0310 14:25:22.361187 141180 round_trippers.go:466] curl -v -XGET -H "X-Csrf-Token: 1" 'https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/.well-known/oauth-authorization-server'
I0310 14:25:22.362381 141180 round_trippers.go:553] GET https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/.well-known/oauth-authorization-server 200 OK in 1 milliseconds
I0310 14:25:22.362423 141180 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 1 ms
I0310 14:25:22.362441 141180 round_trippers.go:577] Response Headers:
I0310 14:25:22.362479 141180 round_trippers.go:580] Date: Fri, 10 Mar 2023 20:25:30 GMT
I0310 14:25:22.362499 141180 round_trippers.go:580] Audit-Id: 544efe6d-a58d-40ff-b749-92e6331029df
I0310 14:25:22.362517 141180 round_trippers.go:580] Cache-Control: no-cache, private
I0310 14:25:22.362536 141180 round_trippers.go:580] Content-Type: application/json
I0310 14:25:22.362554 141180 round_trippers.go:580] Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I0310 14:25:22.362573 141180 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 3d29411f-1f04-4fd6-a3ff-7aa305d1d239
I0310 14:25:22.362592 141180 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: dc5e6c2c-3bd5-41b4-9a4c-5acf0ed9a449
I0310 14:25:22.362611 141180 round_trippers.go:580] Content-Length: 630
I0310 14:25:22.407679 141180 request_token.go:467] falling back to kubeconfig CA due to possible x509 error: x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
I0310 14:25:22.407795 141180 round_trippers.go:466] curl -v -XGET -H "X-Csrf-Token: 1" 'https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net/oauth/authorize?client_id=openshift-challenging-client&code_challenge=sgC5DRN59-kU3iS_4ItafZQsXrcyKDGN3TS2Ymw4Als&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net%2Foauth%2Ftoken%2Fimplicit&response_type=code'
I0310 14:25:22.408715 141180 round_trippers.go:495] HTTP Trace: DNS Lookup for oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net resolved to [{10.4.185.75 }]
I0310 14:25:22.408961 141180 round_trippers.go:510] HTTP Trace: Dial to tcp:10.4.185.75:443 succeed
I0310 14:25:22.416136 141180 round_trippers.go:553] GET https://oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net/oauth/authorize?client_id=openshift-challenging-client&code_challenge=sgC5DRN59-kU3iS_4ItafZQsXrcyKDGN3TS2Ymw4Als&code_challenge_method=S256&redirect_uri=https%3A%2F%2Foauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net%2Foauth%2Ftoken%2Fimplicit&response_type=code in 8 milliseconds
I0310 14:25:22.416176 141180 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 7 ms Duration 8 ms
I0310 14:25:22.416196 141180 round_trippers.go:577] Response Headers:
I0310 14:25:22.416777 141180 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/b05f7d4" 'https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/api/v1/namespaces/openshift/configmaps/motd'
I0310 14:25:22.417886 141180 round_trippers.go:553] GET https://api.csvocp.csv.uschi.nsn-rdnet.net:6443/api/v1/namespaces/openshift/configmaps/motd 403 Forbidden in 1 milliseconds
I0310 14:25:22.417937 141180 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 1 ms
I0310 14:25:22.417955 141180 round_trippers.go:577] Response Headers:
I0310 14:25:22.417977 141180 round_trippers.go:580] Audit-Id: d0693f44-ab60-402d-8b3f-810ff465c2da
I0310 14:25:22.417996 141180 round_trippers.go:580] Cache-Control: no-cache, private
I0310 14:25:22.418015 141180 round_trippers.go:580] Content-Type: application/json
I0310 14:25:22.418034 141180 round_trippers.go:580] X-Content-Type-Options: nosniff
I0310 14:25:22.418053 141180 round_trippers.go:580] Date: Fri, 10 Mar 2023 20:25:30 GMT
I0310 14:25:22.418072 141180 round_trippers.go:580] Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
I0310 14:25:22.418091 141180 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 3d29411f-1f04-4fd6-a3ff-7aa305d1d239
I0310 14:25:22.418112 141180 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: dc5e6c2c-3bd5-41b4-9a4c-5acf0ed9a449
I0310 14:25:22.418131 141180 round_trippers.go:580] Content-Length: 303
I0310 14:25:22.418175 141180 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
error: x509: certificate is valid for api-int.csvocp.csv.uschi.nsn-rdnet.net, not oauth-openshift.apps.csvocp.csv.uschi.nsn-rdnet.net
