Why option rotate in resolv.conf picks up second nameserver as first every time?

Solution Verified - Updated -

Environment

  • All Red Hat Enterprise Linux versions with no local caching application.

Issue

  • Application server tries to query hostname from the list of nameserver's from resolv.conf, the first query always is made to the second nameserver, where it should query our primary server.

Resolution

  • This issue does not have any resolution yet.
  • There is no plan to fix this behaviour in RHEL6 since it is now in Maintenance Support 2 Phase.
  • This issue was resolved in RHEL7.5 with the release of glibc-2.17-222.el7.
  • Red Hat has also filed a request to have man pages updated (RHEL6) with more clear details on how it actually works.

Root Cause

The reason why RES_ROTATE related code picks up the second nameserver every time as the first one is that the item rotation of nameserver list happens before contacting any of the nameservers, thus the second nameserver goes as first every time.

If we have for example the following in resolv.conf:

# cat /etc/resolv.conf 
options rotate
nameserver 192.168.1.5
nameserver 192.168.10.10
nameserver 192.168.20.20
#

The bellow shows the sequence of nameservers in the list before the rotation happens and before any of the nameservers get contacted for the first time - we can see the correct sequence 192.168.1.5, 192.168.10.10, 192.168.20.20:

(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[0])).sin_addr)
$41 = 0x7ffff7fe96d8 "192.168.1.5"
(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[1])).sin_addr)
$42 = 0x7ffff7fe96d8 "192.168.10.10"
(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[2])).sin_addr)
$43 = 0x7ffff7fe96d8 "192.168.20.20"
(gdb)

and the following shows the sequence after the rotation happened and before any of the servers get contacted for the first time - the second defined nameserver 192.168.10.10 will be contacted as first:

(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[0])).sin_addr)
$44 = 0x7ffff7fe96d8 "192.168.10.10"
(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[1])).sin_addr)
$45 = 0x7ffff7fe96d8 "192.168.20.20"
(gdb) p (char *)inet_ntoa((*((struct sockaddr_in *)statp->_u._ext.nsaddrs[2])).sin_addr)
$46 = 0x7ffff7fe96d8 "192.168.1.5"
(gdb)

Diagnostic Steps

If we run the following script under strace:

# cat resolv.py 
import socket
for x in range(5):
   print socket.getaddrinfo('redhat.com', 80);
#

It is showing that the second nameserver defined in resolv.conf gets contacted as first:

# strace -e trace=connect python resolv.py 2>&1 | grep 53
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.10.10")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.20.20")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.1.5")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.10.10")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.20.20")}, 16) = 0
#

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments