SSH connections delay about 20 seconds even if UseDNS is set to no.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 8
  • Red Hat Enterprise Linux (RHEL) 7

Issue

  • The SSH connection is delayed by about 20 seconds, even if the option in the /etc/ssh/sshd_config server configuration is set toUseDNS no.

Resolution

  • Check if sshd, crond, su reporting 'Failed to create session' errors.

    Mar 14 15:21:56 node1.example.com sshd[73161]: pam_systemd(sshd:session): Failed to create session: Connection timed out
    Mar 14 15:30:26 node1.exampl.com crond[120453]: pam_systemd(crond:session): Failed to create session: Connection timed out
    Mar 14 15:32:34 node1.exampl.com login[6928]: pam_systemd(login:session): Failed to create session: Connection timed out

    If so please refer steps in Pam_systemd(crond:session): Failed to create session: Connection timed out

  • Check whether TCP wrappers are configured:

    ~]# cat /etc/hosts.deny
~]# cat /etc/hosts.allow

  • Disable the TCP wrappers if redundant

    ~]# > /etc/hosts.deny
~]# > /etc/hosts.allow
    • If for some reason is necessary to keep the TCP wrappers
      • correct the DNS server addresses
        or
      • make sure you have the names and corresponding IP addresses defined in the /etc/hosts file

  • Configure the firewall using the firewall-cmd command, see Using and configuring firewalld

Root Cause

  • The delay is due to DNS lookups which are timing out. The reason for the lookups are often due to the use of tcp_wrappers through the configuration of the /etc/hosts.allow and /etc/hosts.deny files. These TCP Wrappers files require working DNS resolution.
  • The alternative of tcp-wrappers are firewalls/iptables/nftables useful for filtering network traffic.

Diagnostic Steps

We can see long delays on DNS servers (IP addresses are the same as /etc/resolv.conf):

~]$ cat sshd-strace-from-server
...     
27997 1594711405.553410 open("/etc/hosts.allow", O_RDONLY) = 4
...     
27997 1594711405.553862 open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 6
...     
27997 1594711405.554197 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 6
...     
27997 1594711405.554586 open("/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 6
...     
27997 1594711405.554927 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6
27997 1594711405.554959 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = 0 
27997 1594711405.555003 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}])
27997 1594711405.555039 sendto(6, "\252\357\1\0\0\1\0\0\0\0\0\0\003102\00220\289\00210\7in-add"..., 43, MSG_NOSIGNAL, NULL, 0) = 43 
27997 1594711405.555100 poll([{fd=6, events=POLLIN}], 1, 5000) = 0 (Timeout)
27997 1594711410.560217 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 7
27997 1594711410.560662 connect(7, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = 0 
27997 1594711410.560717 poll([{fd=7, events=POLLOUT}], 1, 0) = 1 ([{fd=7, revents=POLLOUT}])
27997 1594711410.560754 sendto(7, "\252\357\1\0\0\1\0\0\0\0\0\0\003102\00220\289\00210\7in-add"..., 43, MSG_NOSIGNAL, NULL, 0) = 43 
27997 1594711410.560817 poll([{fd=7, events=POLLIN}], 1, 3000) = 0 (Timeout)
27997 1594711413.563939 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 8
27997 1594711413.563981 connect(8, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = 0 
27997 1594711413.564038 poll([{fd=8, events=POLLOUT}], 1, 0) = 1 ([{fd=8, revents=POLLOUT}])
27997 1594711413.564075 sendto(8, "\252\357\1\0\0\1\0\0\0\0\0\0\003102\00220\289\00210\7in-add"..., 43, MSG_NOSIGNAL, NULL, 0) = 43 
27997 1594711413.564160 poll([{fd=8, events=POLLIN}], 1, 6000) = 0 (Timeout)
27997 1594711419.570311 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}])
27997 1594711419.570355 sendto(6, "\252\357\1\0\0\1\0\0\0\0\0\0\003102\00220\289\00210\7in-add"..., 43, MSG_NOSIGNAL, NULL, 0) = 43 
...

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments