RHEL6.6: Linux NFS4 client protocol loop of READ / NFS4ERR_STALE_STATEID followed by RENEW / NFS4_OK with NetApp 9.1.P11

Issue

• Sometime after an NFS server upgrade and reboot, the Linux NFS client is seen having problems such as open() returning EIO and READs to files hanging.
• NFS4 protocol loop as described in https://access.redhat.com/articles/770453 involving READs returning with NFS4ERR_STALE_STATEID followed by RENEW returning with NFS4_OK. A tcpdump at the time of the problem shows a series of READs completing with NFS4ERR_STALE_STATEID and within the same second RENEWs completing with NFS4_OK. The cycle repeats indefinitely.

1139 2018-05-23 14:14:34.980906 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0x6582 Offset:147456 Len:65536  nfs.fh.hash == 0xee7c3e15
1140 2018-05-23 14:14:34.980914 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0x6582 Offset:446464 Len:65536  nfs.fh.hash == 0xee7c3e15
1141 2018-05-23 14:14:34.980916 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 882) READ Status: NFS4ERR_STALE_STATEID
1142 2018-05-23 14:14:34.980921 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 884) READ Status: NFS4ERR_STALE_STATEID
1144 2018-05-23 14:14:34.980945 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0xd1bb Offset:65536 Len:65536  nfs.fh.hash == 0xc8d23fb4
1145 2018-05-23 14:14:34.980953 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0xd1bb Offset:0 Len:65536  nfs.fh.hash == 0xc8d23fb4
1146 2018-05-23 14:14:34.980964 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 889) READ Status: NFS4ERR_STALE_STATEID
1147 2018-05-23 14:14:34.980974 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 892) READ Status: NFS4ERR_STALE_STATEID
1149 2018-05-23 14:14:34.980984 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 893) READ Status: NFS4ERR_STALE_STATEID
1150 2018-05-23 14:14:34.980989 10.1.1.14 -> 10.1.1.200 NFS 174 V4 Call RENEW CID: 0x8872
1151 2018-05-23 14:14:34.981012 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0xbccc Offset:65536 Len:65536  nfs.fh.hash == 0x3a0b9f50
1152 2018-05-23 14:14:34.981014 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 895) READ Status: NFS4ERR_STALE_STATEID
1153 2018-05-23 14:14:34.981027 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 898) READ Status: NFS4ERR_STALE_STATEID
1154 2018-05-23 14:14:34.981030 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0x6582 Offset:425984 Len:4096  nfs.fh.hash == 0xee7c3e15
1155 2018-05-23 14:14:34.981032 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 901) READ Status: NFS4ERR_STALE_STATEID
1157 2018-05-23 14:14:34.981051 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0x6582 Offset:40960 Len:8192  nfs.fh.hash == 0xee7c3e15
1158 2018-05-23 14:14:34.981063 10.1.1.14 -> 10.1.1.200 NFS 242 V4 Call READ StateID:0x9a75 Offset:0 Len:65536  nfs.fh.hash == 0x10409314
...
1386 2018-05-23 14:14:34.983185 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 1139) READ Status: NFS4ERR_STALE_STATEID
1387 2018-05-23 14:14:34.983219 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 1140) READ Status: NFS4ERR_STALE_STATEID
1389 2018-05-23 14:14:34.983230 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 1144) READ Status: NFS4ERR_STALE_STATEID
1390 2018-05-23 14:14:34.983235 10.1.1.200 -> 10.1.1.14 NFS 122 V4 Reply (Call In 1145) READ Status: NFS4ERR_STALE_STATEID
1392 2018-05-23 14:14:34.983268 10.1.1.200 -> 10.1.1.14 NFS 114 V4 Reply (Call In 1150) RENEW