NFS hangs when sysctl tcp_frto=2 due to TCP recovery issues

Issue

• Although the problem was originally reported after a RHEL 5 to RHEL 6 upgrade. This can happen on any RHEL release when "net.ipv4.tcp_frto = 2" which is the default from RHEL6 onwards.
• We have eight rhel 5 servers that we started upgrading to rhel6 (fresh clean install), what we have discovered after upgrading a couple is that in rhel 6, the NFS locks up after a bit of heavier traffic to the nfs.

Jul 14 14:02:05 client kernel: nfs: server server not responding, still trying
Jul 14 14:02:10 client kernel: nfs: server server not responding, still trying
Jul 14 14:02:10 client kernel: nfs: server server not responding, still trying
Jul 14 14:02:38 client kernel: nfs: server  OK
Jul 14 14:02:38 client kernel: nfs: server  OK

We can reproduce this every time. However, we can not duplicate this on rhel 5 servers on the same network. We can duplicate this on different hardware and even a VM. We use the same NFS mount options as we did with RHEL5 etc. etc. etc. but it locks up every time.