RHEL7: NFS clients connected to EMC ISILON NAS hangs periodic 60 seconds during TIME_WAIT state

Solution In Progress - Updated 2017-09-06T19:26:51+00:00 -

Issue

NFS Server randomly hangs 60 seconds. The NFS server is EMC ISILON, and shares are using TCP. UDP shares are not affected.
NFS clients become unresponsive, with "nfs or lockd server not responding" messages appearing for each RPC request made while the remote endpoint is at TIME_WAIT state.
During reuse of TCP 4-tuple, NFS clients locks up tcp retransmists for upto a minute.

Seen with EMC ISILON NFS server
(isilon running OneFS 8.0.0.4 and older)
Red Hat Enterprise Linux (RHEL) 6.9 (NFS client)
kernel 2.6.32-696.el6 and above
Red Hat Enterprise Linux 7 (NFS client)
kernels 3.10.0-514.el7 and above
NFS over TCP, TCP timestamps option enabled
net.ipv4.tcp_timestamps = 1
Only the kernel versions that attempts to reuse the same source port.
Please refer this article for the kernel versions that reuse the same source port RHEL6 / RHEL7: After an NFS client's TCP connection goes idle, does it reconnect using the same source port or a different port?

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.