Authentication and Openshift-Apiserver operators are degraded with error " WellKnownReadyController_SyncError " in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform
    • 4
  • VMware vSphere

Issue

  • Authentication Operator fails with error:

    message: 'WellKnownReadyControllerDegraded: failed to GET kube-apiserver oauth
      endpoint https://10.x.x.x:6443/.well-known/oauth-authorization-server:
      Tunnel or SSL Forbidden'
    reason: WellKnownReadyController_SyncError
    status: "True"
    type: Degraded
    
    
  • Openshift-apiserver fails with error:

    message: 'APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable
      for apiserver.openshift-apiserver (crashlooping container is waiting in apiserver-58c7b57f98-5l4vc
      pod)'
    reason: APIServerDeployment_UnavailablePod
    status: "True"
    type: Degraded
    

Resolution

  • Edit the proxy configuration:

    $ oc edit proxy/cluster
    
  • Check the noproxy parameter in the spec section:

    spec:
    httpProxy: http://<ip>:80
    httpsProxy: http://<ip>:80
    noProxy: example.com
    
  • Add machine CIDR and vCenter IP to noproxy parameter:

Root Cause

noproxy should contain a list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying. One must include vCenter’s IP address and the IP range that is used for its machines.

Diagnostic Steps

  • Login to the OAuth pod and check if you can reach API through proxy:

    $ oc rsh <oauth pod> 
    $ curl -kv https://$API:6443/.well-known/oauth-authorization-server
    
  • Authentication and openshift-apiserver are degraded true:

    $ oc get co 
    
    NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    authentication                             4.15.15   False       False         True       16h
    openshift-apiserver                        4.15.15   True        True          True       14h
    
  • One of the openshift-apiserver is in CrashLoopBackOff:

    $ oc get pods -n openshift-apiserver
    
    NAME                         READY   STATUS             RESTARTS   AGE
    apiserver-58c7b57f98-xxxx   1/2     CrashLoopBackOff   162        12h
    apiserver-59946d7678-xxx5   2/2     Running            0          13h
    apiserver-878d86d76-lxxxx    2/2     Running            0          13h
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments