Troubleshooting CanaryChecksRepetitiveFailures in the Ingress operator on OpenShift 4

Resolution

The CanaryChecksRepetitiveFailures error can be caused by different reasons, and some of them have been identified as bugs and resolved in different releases. Refer to the following solutions to check if the issue is caused by any of the already known issues (there could be other solutions also related to the same error):

• Ingress Operator degraded with CanaryChecksRepetitiveFailures.
• Ingress Operator failed: CanaryChecksRepetitiveFailures during upgrade when ingress sharding is used.
• Canary route checks failing or canary route deployed with two routerCanonicalHostname entries with different domains.
• Failed to update canary route openshift-ingress-canary/canary in OpenShift 4.
• CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing in OpenShift 4.17.

If none of the above is the cause of the current issue, the following steps outline a methodology for diagnosing the problem and finding the root cause of the problem.

Performing each step below is designed to illustrate where in the communication flow there might be an issue, to help you then further identify and resolve the root cause for that outage. These steps are designed to show you where there is a breakdown.

The canary route check repetitive failures error explicitly is reporting that the following connection flow is failing:

openshift-ingress-operator pod --> DNS lookup for *.apps (A-record for loadbalancer) --> Loadbalancer IP --> router-default pod --> openshift-ingress-canary-pod (200 OK).

Troubleshooting steps

If one of the portions of the curl request fails or is failing, this error will be shown. The below steps will help to find which part of that flow is broken in order to solve the issue:

1. Test if pods can curl itself for port 8080. All curl request should provide 200 status code, if status code != 200, the pod is not yet operational:

   $ for i in `oc get po  -n openshift-ingress-canary | grep -vi name | awk '{print $1}' ` ; do echo -e "\t---- $i ----" ; oc exec -n openshift-ingress-canary $i  --  curl -s -D - http://localhost:8080/ ; done
   

   Note: in OCP 4.17 and newer releases, the curl in above command needs to be changed to curl -sk -D - https://localhost:8888/

   • If issue is recent, please wait for couple of minutes (1 or 2) and check again.
   • If issue is persistent, restart pod.

2. Check if pods can curl on service end-point. All curl request should provide 200 status code, if status code != 200, service IP is not able to route incoming traffic.

   $ SVC_IP=$(oc get svc -n openshift-ingress-canary -ojsonpath={..clusterIP})
   $ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- curl -s -D - http://${SVC_IP}:8080 ; done
   

   Note: in OCP 4.17 and newer releases, the curl in above command needs to be changed to curl -sk -D - https://${SVC_IP}:8888

   • Creating this service is managed by daemonset ingress-canary in openshift-ingress-canary namespace, if status code is != 200, restart cluster-version-operator pod and then mentioned daemonset.

3. Check connection to route. The curl request should give 302 response code:

   $ ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
   $ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- curl http://${ROUTE} -sS -k -D - ; done
   

   • Check if load-balancer is routing traffic to router pods.
   • If route is not present or status code is != 302, restart the cluster-version operator -> router pods from openshift-ingress namespace.

4. Confirm DNS routing is resolving properly from within the openshift-ingress-operator pod:

   $ ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
   $ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i -- dig ${ROUTE} +nocmd +noall +answer ; done
   ##it should return an A record that points to the IP of the loadbalancer for *.apps.<yourcluster>.<yourdomain>
   

   • A failure here may indicate that DNS is not working locally on the platform (openshift-dns pods); or that the upstream nameserver is failing to return results for *.apps...

5. Confirm the Load Balancer is forwarding traffic appropriately by using curl --resolve to bypass the LB and terminate the lookup directly at a router pod from your bastion:

   $ oc get pods -o wide -n openshift-ingress
   NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE                                                      
   router-default-5fc6f8b969-jhcnd   1/1     Running   0          9h    10.0.90.154   worker-2 
   router-default-5fc6f8b969-kfq45   1/1     Running   0          19h   10.0.89.132   worker-0 
   
   $ ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
   $ ROUTER=$(oc get pod -n openshift-ingress -o wide | grep -v NAME | grep Running | grep router-default | awk {'print $6'} | head -n 1)
   $ curl -k -v --noproxy '*' --resolve ${ROUTE}:443:${ROUTER} https://${ROUTE}
   

   Note: in this example, ${ROUTER} can be 10.0.89.132 or 10.0.90.154, we are selecting the first available router pod that is in Running state for this test

   Note: *when copying out the curl command to your command line, it may change the syntax of the curl to include backslashes: https://$\{ROUTE\} - remove these, as it will fail the curl in error otherwise.

   • Check in the curl result that the syntax includes confirmation that you did indeed resolve at the router pod and NOT the Load Balancer IP: (note that the result below indicates 10.0.90.154 was called, which is the IP of the router pod/infra host)

     * Connected to canary-openshift-ingress-canary.apps.<mycluster>.<mydomain> (10.0.90.154) port 443 (#0)
     

   • We should see that the router pods are able to resolve the request and return a valid response code from this route. A success here may imply that your Loadbalancer is not able to forward traffic, or that the DNS nameserver is not resolving properly at the ingress VIP A record (*.apps.yourcluster.yourdomain) to the Loadbalancer IP.

   • After several minutes, if the ingress is still degraded try the following command:

     $ oc delete route canary -n openshift-ingress-canary
     

NOTE: if issue is persistent after executing above steps, please open support case with the curl outputs, dig results and up-to-date must-gather.