Troubleshooting Red Hat OpenShift Container Platform 4: DNS

Diagnostic Steps

• Check the cluster operator to see if it is available:

  $ oc get clusteroperator dns
  

• Check to see if there are pods and services created under openshift-dns-operator:

  $ oc -n openshift-dns-operator get all -o wide
  

• Check the logs of dns-operator pod:

  $ oc logs pod/`oc get pods -o=jsonpath="{.items[0].metadata.name}" -n openshift-dns-operator` -n openshift-dns-operator 2>/dev/null || oc logs pod/`oc get pods -o=jsonpath="{.items[0].metadata.name}" -n openshift-dns-operator` -c dns-operator -n openshift-dns-operator
  

• Check dns components are running in the openshift-dns project:

  $ oc -n openshift-dns get all
  

• Check the pod resolver is pointing to DNS Service IP:

  $ export PODS=`oc get pods -o=jsonpath="{.items[*].metadata.name}" -n openshift-apiserver`; for pod in $PODS; do oc exec $pod -c openshift-apiserver -n openshift-apiserver -- cat /etc/resolv.conf ; done;
  

• Check the coredns container logs:

  $ export PODS=`oc get pods -o=jsonpath="{.items[*].metadata.name}" -n openshift-dns -l dns.operator.openshift.io/daemonset-dns=default`; for pod in $PODS; do oc logs $pod -c dns -n openshift-dns|sed "s/^/$pod\t/"; echo; done
  

• Verify that both TCP and UDP requests from the coredns container to the upstream DNS server are possible. Both TCP and UDP connections to the upstream DNS server are required for CoreDNS to function correctly. Replace [upstream_dns_IP] and [upstream_dns_port] with the correct upstream IP and port:

  $ UPSTREAM_DNS_IP="[upstream_dns_IP]"; UPSTREAM_DNS_PORT="[upstream_dns_port]"; echo -e "\nTCP\n"; for dnspod in `oc get pods -n openshift-dns -o name --no-headers -l dns.operator.openshift.io/daemonset-dns=default`; do echo "Pod $dnspod"; oc exec -n openshift-dns -c dns $dnspod -- dig @${UPSTREAM_DNS_IP} redhat.com -p ${UPSTREAM_DNS_PORT} +tcp +short; echo; done
  
  $ UPSTREAM_DNS_IP="[upstream_dns_IP]"; UPSTREAM_DNS_PORT="[upstream_dns_port]"; echo -e "\nUDP\n"; for dnspod in `oc get pods -n openshift-dns -o name --no-headers -l dns.operator.openshift.io/daemonset-dns=default`; do echo "Pod $dnspod"; oc exec -n openshift-dns -c dns $dnspod -- dig @${UPSTREAM_DNS_IP} redhat.com -p ${UPSTREAM_DNS_PORT} +notcp +short; echo; done
  

• Test resolving kubernetes service hostname to and from every DNS pod:

  $ DST_HOST=kubernetes.default.svc.cluster.local; for dnspod in `oc get pods -n openshift-dns -o name --no-headers -l dns.operator.openshift.io/daemonset-dns=default`; do for dnsip in `oc get pods -n openshift-dns -l dns.operator.openshift.io/daemonset-dns=default -o go-template='{{ range .items }} {{index .status.podIP }} {{end}}'`; do echo -ne "$dnspod\tquerying $DST_HOST to $dnsip ->\t"; oc exec -n openshift-dns $dnspod -- dig @$dnsip $DST_HOST -p 5353 +short 2>/dev/null ; done; done
  

• Test resolving external queries like redhat.com:

  $ DST_HOST=redhat.com; for dnspod in `oc get pods -n openshift-dns -o name --no-headers -l dns.operator.openshift.io/daemonset-dns=default`; do for dnsip in `oc get pods -n openshift-dns -o go-template='{{ range .items }} {{index .status.podIP }} {{end}}' -l dns.operator.openshift.io/daemonset-dns=default`; do echo -ne "$dnspod\tquerying $DST_HOST to $dnsip ->\t"; oc exec -n openshift-dns $dnspod -- dig @$dnsip $DST_HOST -p 5353 +short 2>/dev/null ; done; done
  

  Important: If there are DNS forwarding entries in the dns operator, pay special attention to run these commands for addresses that will only resolve with that forwarding.
  Also please note that image lookups will not use the DNS operator.

• Check within a pod to gather the DNS lookup time versus total request time (change the [namespace_name] and [pod_name] with the name of a namespace and a pod):

  • IPv4 and IPv6

    $ oc exec -n [namespace_name] [pod_name] -- bash -c 'while true; do echo -n "$(date)  "; curl -s -o /dev/null -w "%{time_namelookup} %{time_total} %{http_code}\n" https://www.redhat.com -k; sleep 10; done'
    

  • IPv4 only:

    $ oc exec -n [namespace_name] [pod_name] -- bash -c 'while true; do echo -n "$(date)  "; curl -s -o /dev/null -w "%{time_namelookup} %{time_total} %{http_code}\n" --ipv4 https://www.redhat.com -k; sleep 10; done'
    

  • IPv6 only

    $ oc exec -n [namespace_name] [pod_name] -- bash -c 'while true; do echo -n "$(date)  "; curl -s -o /dev/null -w "%{time_namelookup} %{time_total} %{http_code}\n" --ipv6 https://www.redhat.com -k; sleep 10; done'
    

• From inside a pod, run the following commands to potentially isolate SDN connections issues towards the coredns pods (change the [namespace_name] and [pod_name] with the name of a namespace and a pod):

  $ for pod in $(oc -n openshift-dns get pod -l dns.operator.openshift.io/daemonset-dns -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}'); do oc exec -n [namespace_name] [pod_name] -- nslookup -port=5353 www.redhat.com $pod; done