FD_SOCK Server Socket Is Never Closed After Network Interruption

Issue

• We have a TCP based cluster. We notice that when looking at the open connections on each server (via netstat -an), the server listening on the 57600 port range shows a very large number of connections established by the other server; however, when we look at the netstat on the other server, it shows little to no established connections with the first server on ports in the 57600 range (i.e., there are no connections to server1:5760X listed in the netstat output. These ports are in the FD_SOCK port range. Here is a sample output:

netstat from 1.2.3.4:

tcp 0 0 1.2.3.4:57600 10.20.30.40:53740 ESTABLISHED
tcp 0 0 1.2.3.4:57603 10.20.30.40:43769 ESTABLISHED
tcp 0 0 1.2.3.4:57600 10.20.30.40:59069 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:35035 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:53479 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:38461 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:43358 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:33445 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:36477 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:37472 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:47168 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:58110 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:57735 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:39453 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:51956 ESTABLISHED
tcp 0 0 1.2.3.4:57600 10.20.30.40:38552 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:52788 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:48483 ESTABLISHED
tcp 0 0 1.2.3.4:57600 10.20.30.40:33573 ESTABLISHED
tcp 0 0 1.2.3.4:57600 10.20.30.40:34312 ESTABLISHED
tcp 0 0 1.2.3.4:57600 10.20.30.40:40308 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:55021 ESTABLISHED
tcp 0 0 1.2.3.4:57604 10.20.30.40:36513 ESTABLISHED
tcp 0 0 1.2.3.4:57603 10.20.30.40:39913 ESTABLISHED
tcp 0 0 1.2.3.4:57602 10.20.30.40:37743 ESTABLISHED
tcp 0 0 1.2.3.4:57601 10.20.30.40:41677 ESTABLISHED

netstat from 10.20.30.40:

tcp 0 0 10.20.30.40:43358 1.2.3.4:57602 ESTABLISHED
tcp 0 0 10.20.30.40:57735 1.2.3.4:57601 ESTABLISHED
tcp 0 0 10.20.30.40:51956 1.2.3.4:57604 ESTABLISHED
tcp 0 0 10.20.30.40:40308 1.2.3.4:57600 ESTABLISHED