NFS4 client hung trying to open files against NetApp NFS4 server, waiting for nfs4_run_state_manager to complete

Issue

• Starting last night at around 10:30pm it looks like the server started having problems. Currently there are 154 processes which are blocking on I/O and are essentially hung most likely on NFS but I can’t determine which NFS filesystem as there’s many mounted. The easiest solution is to reboot the box.
• Many processes trying to open files on NFS4, waiting in nfs4_wait_clnt_recover, for the NFS4 state manager thread to complete recovery of open state
• Other processes blocked on a filesystem mutex, also trying to open files. The mutex is held by the other process waiting on the state manager thread.
• The NFS4 state manager thread is running and seems to be waiting for an OPEN to complete, call tree: nfs4_do_reclaim ... nfs4_open_expired ... nfs4_run_open_task...
• NFS4 client sending OPENs to the same file over and over in rapid fire