How do I reduce the time a socket spends in the TIME-WAIT state
Environment
- Red Hat Enterprise Linux 5 and 6
- TCP timestamps are disabled
Issue
- Site uses load balancing hardware which does not support the TCP time stamp option
- The connection rate is large enough that port numbers get reused quickly, somethimes within a minute
- If a connection is made using a port number that is still associated with a socket in the TIME-WAIT state and the initial sequence number of the new connection is less than the current sequence number for the socket the result to the connection request is not to established the connection (The SYN segment from the client is ACKed using the old sequence numbers instead of a SYN-ACK using new sequence numbers)
Resolution
- By using this script you are effectively short circuiting the way TCP is suppose to work.
- Install scapy first (Scapy can be found in the epel repo).
# yum install scapy
- The following scapy/python script can be used to terminate sockets in a TIME-WAIT state -- please read all the disclaimers
# !/usr/bin/env python
#
#
# Disclaimer: The following information has been provided by Red Hat, but is outside the scope of the posted
# [Service Level Agreements] (https://www.redhat.com/support/service/sla/) and support procedures. The information is
# provided as-is and any configuration settings or installed applications made from the information in this article could
# make the Operating System unsupported by Red Hat Global Support Services. The intent of this article is to provide
# information to accomplish the system's needs. Use of the information in this article at the user's own risk.
#
# The purpose of this script is to spoof an ACK from the source IP to the destination to trigger a duplicate ACK packet
# from the destination back to the source which will then, we hope, send a reset. This script cannot be run on the
# destination host but it doesn't have to be run on the source host (as long as firewall rules and other settings permit
# the spoofed packet to be sent).
#
# You can use the command
#
# ssh root@DESTINATION "ss -nao state TIME-WAIT | grep -v Recv-Q" | awk '{print $3 ":" $4}' | awk 'BEGIN {FS=":"} { print $1 " " $2 " " $3 " " $4 " " $5}' | while read IPDst TCPDst IPSrc TCPSrc ; do echo `date` $IPSrc $TCPSrc $IPDst $TCPDst; ./trigger_dupACK.py em1 $IPSrc $TCPSrc $IPDst $TCPDst; done
#
# where the first part has ssh execute the ss command for TIME-WAIT sockets and strips off the header. The "awk" part
# formats the lines properly and then the "while" loop reads the lines and executes the trigger_dupACK.py script
#
# Obviously you can change or add other filters to the ss command or after the data is returned. For example
#
# ssh root@DESTINATION "ss -nao state TIME-WAIT" | grep -e "timewait,4[43210]" -e "timewait,[321]" | awk '{print $3 ":" $4}' | awk 'BEGIN {FS=":"} {print $1 " " $2 " " $3 " " $4 " " $5}' | while read IPDst TCPDst IPSrc TCPSrc ; do echo `date` $IPSrc $TCPSrc $IPDst $TCPDst; ./trigger_dupACK.py em1 $IPSrc $TCPSrc $IPDst $TCPDst; done
#
# will select only TIME-WAIT-sockets with a timer at 44 seconds or less, in effect allowing for the TIME-WAIT socket to exist
# for at least 15 seconds (60-45) but killing it after that.
#
# or
# ssh root@DESTINATION "ss -nao state FIN-WAIT-2" | grep -v Recv-Q" | awk '{print $3 ":" $4}' | awk 'BEGIN {FS=":"} {print $1 " " $2 " " $3 " " $4 " " $5}' | while read IPDst TCPDst IPSrc TCPSrc ; do echo `date` $IPSrc $TCPSrc $IPDst $TCPDst; ./trigger_dupACK.py em1 $IPSrc $TCPSrc $IPDst $TCPDst; done
#
# will select all the sockets in a FIN-WAIT-2 state
#
# It may be necessary to change the rp_filter setting on destination to 0 or 2 and/or firewall rules to allow the spoofed
# packet. It may also be necessary to set the net.ipv4.tcp_rfc1337 back to its default value of 0 on destination. If the value
# is not 0 the reset will be ignored and the timer on the socket reset. This will result in the socket never timing out if the
# script is run in a loop with a cycle time less than 60 seconds.
#
# If you are going to run this in some kind of loop and it will be using ssh you will need to configure keys as an access
# method otherwise it will keep prompting you for your password.
#
# This script must be run as root for scapy to send the packet but it is not necessary to log into the destination as root.
#
# Note that not all hosts will send a reset when they get the ACK. It may be that since they no longer have a connection
# their firewall drops the segment. The issue is that the spoofed ACK resets the timewait timer so instead of killing the
# socket the script effectively keeps it active forever. The only thing you can do is filter out those hosts once they are
# identified.
# For example if 10.20.30.40 where such a host
#
# ssh root@DESTINATION "ss -nao state TIME-WAIT" | grep -v 10.20.30.40 | grep -e "timewait,4[543210]" -e "timewait,[321]" | awk '{print $3 ":" $4}' | awk 'BEGIN {FS=":"} {print $1 " " $2 " " $3 " " $4 " " $5}' | while read IPDst TCPDst IPSrc TCPSrc ; do echo date $IPSrc $TCPSrc $IPDst $TCPDst; ./trigger_dupACK.py em1 $IPSrc $TCPSrc $IPDst $TCPDst; done
#
#
# This script uses scapy which requires python. Scapy can be installed from the epel repo. While the script should run
# without problems are links for scapy documentation and packet crafting
# [Scapy v2.1.1-dev documentation] (http://www.secdev.org/projects/scapy/doc/installation.html)
# [Packet Crafting using Scapy] (http://www.scs.ryerson.ca/~zereneh/cn8001/CN8001-PacketCraftingUsingScapy-WilliamZereneh.pdf)
# [The very unofficial Dummies Guideto Scapy ] (http://theitgeekchronicles.files.wordpress.com/2012/05/scapyguide1.pdf)
# [network packet forgery with scapy ] (http://www.secdev.org/conf/scapy_pacsec05.handout.pdf)
# There are lots of others as well, google is your friend.
# Usage
# ./trigger_dupACK.py device source-IP source-TCP dest-IP dest-TCP
# where
# device device that will send the packet
# source-IP source IP address of the packet
# source-TCP source TCP port of the packet
# dest-IP destination IP adress of the packet
# dest-TCP destination TCP port of the packet
import logging
logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
from scapy.all import*
Dev = sys.argv[1]
IPSrc = sys.argv[2]
TCPSrc = int(sys.argv[3])
IPDst = sys.argv[4]
TCPDst = int(sys.argv[5])
TCPSEQ = 0xfedc6789
TCPflags = "PA"
pak=IP(src=IPSrc, dst=IPDst, id=TCPSrc)/TCP(sport=TCPSrc, dport=TCPDst, flags=TCPflags, seq=TCPSEQ, ack=TCPDst)
send(pak, iface=Dev, verbose=0)
#
# end trigger_dupACK.py
Root Cause
TCP is behaving correctly
Diagnostic Steps
Use "netstat -an|grep TIME" to see sockets in the TIME_WAIT state.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
