Authentication and Openshift-Apiserver operators are degraded with error " WellKnownReadyController_SyncError " in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform
    • 4
  • VMware vSphere

Issue

  • Authentication Operator fails with error:

    message: 'WellKnownReadyControllerDegraded: failed to GET kube-apiserver oauth
  endpoint https://10.x.x.x:6443/.well-known/oauth-authorization-server:
  Tunnel or SSL Forbidden'
reason: WellKnownReadyController_SyncError
status: "True"
type: Degraded

  • Openshift-apiserver fails with error:

    message: 'APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable
  for apiserver.openshift-apiserver (crashlooping container is waiting in apiserver-58c7b57f98-5l4vc
  pod)'
reason: APIServerDeployment_UnavailablePod
status: "True"
type: Degraded

Resolution

  • Edit the proxy configuration:

    $ oc edit proxy/cluster

  • Check the noproxy parameter in the spec section:

    spec:
httpProxy: http://<ip>:80
httpsProxy: http://<ip>:80
noProxy: example.com

  • Add machine CIDR and vCenter IP to noproxy parameter:

Root Cause

noproxy should contain a list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying. One must include vCenter’s IP address and the IP range that is used for its machines.

Diagnostic Steps

  • Login to the OAuth pod and check if you can reach API through proxy:

    $ oc rsh <oauth pod> 
$ curl -kv https://$API:6443/.well-known/oauth-authorization-server

  • Authentication and openshift-apiserver are degraded true:

    $ oc get co 

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.15.15   False       False         True       16h
openshift-apiserver                        4.15.15   True        True          True       14h

  • One of the openshift-apiserver is in CrashLoopBackOff:

    $ oc get pods -n openshift-apiserver

NAME                         READY   STATUS             RESTARTS   AGE
apiserver-58c7b57f98-xxxx   1/2     CrashLoopBackOff   162        12h
apiserver-59946d7678-xxx5   2/2     Running            0          13h
apiserver-878d86d76-lxxxx    2/2     Running            0          13h

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments