Upgrade process fails in Openshift Container Platform 4 because the Ingress Operator failed: CanaryChecksRepetitiveFailures

Resolution

1. Test if pods can curl itself for port 8080. All curl request should provide 200 status code, if status code != 200, the pod is not yet operational.

   # for i in `oc get po  -n openshift-ingress-canary | grep -vi name | awk '{print $1}' ` ; do echo -e "\t---- $i ----" ; oc exec -n openshift-ingress-canary $i  --  curl -s -D - http://localhost:8080/ ; done
   

   • If issue is recent, please wait for couple of minutes (1 or 2) and check again.
   • If issue is persistent, restart pod.

2. Check if pods can curl on service end-point. All curl request should provide 200 status code, if status code != 200, service IP is not able to route incoming traffic.

   # SVC_IP=$(oc get svc -n openshift-ingress-canary -ojsonpath={..clusterIP})
   # for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- curl http://${SVC_IP}:8080 -s -D - ; done
   

   • Creating this service is managed by daemonset ingress-canary in openshift-ingress-canary namespace, if status code is != 200, restart cluster-version-operator pod and then mentioned daemonset.

3. Check connection to route. Curl request should give 302 response code.

   # ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
   # for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- curl http://${ROUTE} -sS -k -D - ; done
   

   • Check if load-balancer is routing traffic to router pods.
   • If route is not present or status code is != 302, restart cluster-version operator -> router pods from openshift-ingress namespace.

4. Confirm DNS routing is resolving properly from within the openshift-ingress-operator pod:

   # ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
   # for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- dig ${ROUTE} +nocmd +noall +answer ; done
   ##you should return an A record that points to the IP of your loadbalancer for *.apps.<yourcluster>.<yourdomain>
   

   • A failure here may indicate that DNS is not working locally on the platform (openshift-dns pods); or that the upstream nameserver is failing to return results for *.apps...

5. Confirm your Loadbalancer is forwarding traffic appropriately by using curl --resolve to bypass the LB and terminate the lookup directly at a router pod from your bastion:

   $ oc get pods -o wide -n openshift-ingress
   NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                                      
   router-default-5fc6f8b969-jhcnd   1/1     Running   0          9h    10.0.90.154   worker-2 
   router-default-5fc6f8b969-kfq45   1/1     Running   0          19h   10.0.89.132   worker-0 
   
   ##while on bastion run the following syntax to 
   $ curl -kv --resolve {ROUTE-URL}:80:{INFRA-NODE-IP-ADDRESS} http://{ROUTE-URL}
   $ curl -kv --resolve {ROUTE-URL}:443:{INFRA-NODE-IP-ADDRESS} https://{ROUTE-URL}
   

   Note: In this example, {INFRA-NODE-IP-ADDRESS} can be 10.0.89.132 or 10.0.90.154

   • Check in your curl result that the syntax includes confirmation that you did indeed resolve at the router pod:

   * Connected to canary-openshift-ingress-canary.apps.<mycluster>.<mydomain> (10.0.90.154) port 443 (#0)
   

   • We should see that the router pods are able to resolve the request and return a valid response code from this route. A success here may imply that your Loadbalancer is not able to forward traffic, or that the DNS nameserver is not resolving properly at the ingress VIP A record (*.apps.yourcluster.yourdomain) to the Loadbalancer IP.

If issue is persistent after executing above steps, please open support case with the curl outputs, dig results and up-to-date must-gather.