Intermittent dns query failures when forwarding requests from dnsmasq

Issue

• dnsmasq uses low source port for dns queries.
• Intermittent dns query failures/drops when forwarding requests from dnsmasq
• UnknownHostException randomly from an OpenShift node with dnsmasq
• After enabling query logs in dnsmasq, it received a query for IPv4 and IPv6 FQDN but received no reply from SkyDNS. Each request was sent to SkyDNS twice

$ grep 'Sep 25 19:13:24' dnsmasq.node1.log. | grep example.test
Sep 25 19:13:24 dnsmasq[56364]: 136257 100.72.10.157/52783 query[A] test.example.svc.cluster.local from 100.72.10.157
Sep 25 19:13:24 dnsmasq[56364]: 136257 100.72.10.157/52783 forwarded test.example.svc.cluster.local to 127.0.0.1
Sep 25 19:13:24 dnsmasq[56364]: 136257 100.72.10.157/52783 forwarded test.example.svc.cluster.local to 127.0.0.1
Sep 25 19:13:24 dnsmasq[56364]: 136258 100.72.10.157/52783 query[AAAA] test.example.svc.cluster.local from 100.72.10.157
Sep 25 19:13:24 dnsmasq[56364]: 136258 100.72.10.157/52783 forwarded test.example.svc.cluster.local to 127.0.0.1
Sep 25 19:13:24 dnsmasq[56364]: 136258 100.72.10.157/52783 forwarded test.example.svc.cluster.local to 127.0.0.1

• The SkyDNS logs (after setting atomic-openshift-node service logs to loglevel 4) suggest it responded

$ grep '2018-09-25T19:13:24' atomic-openshift-services.node1.log. | grep example.test
2018-09-25T19:13:24.008936-04:00 node1 atomic-openshift-node: I0925 19:13:24.008725   48957 logs.go:41] skydns: received DNS Request for "test.example.svc.cluster.local." from "127.0.0.1:53" with type 1
2018-09-25T19:13:24.009163-04:00 node1 atomic-openshift-node: I0925 19:13:24.008775   48957 serviceresolver.go:88] Answering query test.example.svc.cluster.local.:false
2018-09-25T19:13:24.009362-04:00 node1 atomic-openshift-node: I0925 19:13:24.008822   48957 logs.go:41] skydns: received DNS Request for "test.example.svc.cluster.local." from "127.0.0.1:53275" with type 28
2018-09-25T19:13:24.009542-04:00 node1 atomic-openshift-node: I0925 19:13:24.008806   48957 serviceresolver.go:161] Answered test.example.svc.cluster.local.:false with msg.Service{Host:"100.125.3.44", Port:0, Priority:10, Weight:10, Text:"", Mail:false, Ttl:0x1e, TargetStrip:0, Group:"", Key:"/skydns/local/cluster/svc/test/example/6bd0a10f"}
2018-09-25T19:13:24.009754-04:00 node1 atomic-openshift-node: I0925 19:13:24.008846   48957 serviceresolver.go:88] Answering query test.example.svc.cluster.local.:false
2018-09-25T19:13:24.009936-04:00 node1 atomic-openshift-node: I0925 19:13:24.008867   48957 logs.go:41] skydns: received DNS Request for "test.example.svc.cluster.local." from "127.0.0.1:53275" with type 28
2018-09-25T19:13:24.010120-04:00 node1 atomic-openshift-node: I0925 19:13:24.008865   48957 serviceresolver.go:161] Answered test.example.svc.cluster.local.:false with msg.Service{Host:"100.125.3.44", Port:0, Priority:10, Weight:10, Text:"", Mail:false, Ttl:0x1e, TargetStrip:0, Group:"", Key:"/skydns/local/cluster/svc/test/example/6bd0a10f"}
2018-09-25T19:13:24.010301-04:00 node1 atomic-openshift-node: I0925 19:13:24.008887   48957 logs.go:41] skydns: received DNS Request for "test.example.svc.cluster.local." from "127.0.0.1:53" with type 1
2018-09-25T19:13:24.010477-04:00 node1 atomic-openshift-node: I0925 19:13:24.008954   48957 serviceresolver.go:88] Answering query test.example.svc.cluster.local.:false
2018-09-25T19:13:24.010704-04:00 node1 atomic-openshift-node: I0925 19:13:24.008970   48957 serviceresolver.go:161] Answered test.example.svc.cluster.local.:false with msg.Service{Host:"100.125.3.44", Port:0, Priority:10, Weight:10, Text:"", Mail:false, Ttl:0x1e, TargetStrip:0, Group:"", Key:"/skydns/local/cluster/svc/test/example/6bd0a10f"}
2018-09-25T19:13:24.010912-04:00 node1 atomic-openshift-node: I0925 19:13:24.008889   48957 serviceresolver.go:88] Answering query test.example.svc.cluster.local.:false
2018-09-25T19:13:24.011096-04:00 node1 atomic-openshift-node: I0925 19:13:24.009040   48957 serviceresolver.go:161] Answered test.example.svc.cluster.local.:false with msg.Service{Host:"100.125.3.44", Port:0, Priority:10, Weight:10, Text:"", Mail:false, Ttl:0x1e, TargetStrip:0, Group:"", Key:"/skydns/local/cluster/svc/test/example/6bd0a10f"}