RHEL5: nfs server hangs or hung_task panic, while accessing files on pmfs filesystem

Issue

• nfsd threads stuck in backtraces waiting on uvfs_make_request, similar to

PID: 17735  TASK: ffff8103ef8c67b0  CPU: 1   COMMAND: "nfsd"
 #0 [ffff81010fe0bb00] schedule at ffffffff80062fa0
 #1 [ffff81010fe0bbd8] uvfs_make_request at ffffffff8874d4ea [pmfs]
 #2 [ffff81010fe0bc38] uvfs_lookup_by_name at ffffffff8874c0d5 [pmfs]
 #3 [ffff81010fe0bc78] uvfs_dentry_revalidate at ffffffff8874c1c6 [pmfs]
 #4 [ffff81010fe0bd08] __lookup_hash at ffffffff800371ce
 #5 [ffff81010fe0bd48] lookup_one_len at ffffffff800edb09
 #6 [ffff81010fe0bd68] nfsd_lookup_dentry at ffffffff88780b1b [nfsd]
 #7 [ffff81010fe0bdd8] nfsd_lookup at ffffffff88780be2 [nfsd]
 #8 [ffff81010fe0be18] nfsd_proc_lookup at ffffffff8877dc39 [nfsd]
 #9 [ffff81010fe0be48] nfsd_dispatch at ffffffff8877c1db [nfsd]
#10 [ffff81010fe0be78] svc_process at ffffffff885db83e [sunrpc]
#11 [ffff81010fe0bee8] nfsd at ffffffff8877c770 [nfsd]
#12 [ffff81010fe0bf48] kernel_thread at ffffffff8005dfc1

• a hung_task backtrace and/or panic may be triggered eventually, due to another nfsd thread waiting on an inode mutex, held by one of the processes waiting on uvfs_make_request

INFO: task nfsd:17740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D ffff81000900c9a0     0 17740      1         17739 17733 (L-TLB)
 ffff810396dd9d00 0000000000000046 ffff810396dd9d60 ffffffff80077b6e
 0000000000000008 000000000000000a ffff8103743c2040 ffff81010ef4a0c0
 000292afccc20ccd 000000000001307e ffff8103743c2228 000000018874c958
Call Trace:
 [<ffffffff80077b6e>] smp_send_reschedule+0x4e/0x53
 [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
 [<ffffffff80063aff>] mutex_lock+0x13/0x1d
 [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
 [<ffffffff88780aab>] :nfsd:nfsd_lookup_dentry+0x32b/0x440
 [<ffffffff885de684>] :sunrpc:ip_map_match+0x19/0x30
 [<ffffffff88780be2>] :nfsd:nfsd_lookup+0x22/0xb0
 [<ffffffff885de76e>] :sunrpc:ip_map_lookup+0xbc/0xc3
 [<ffffffff8877dc39>] :nfsd:nfsd_proc_lookup+0x7c/0x98
 [<ffffffff8877c1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
 [<ffffffff885db83e>] :sunrpc:svc_process+0x3f8/0x6bf
 [<ffffffff80064624>] __down_read+0x12/0x92
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8877c770>] :nfsd:nfsd+0x1a5/0x2c9
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8005dfb7>] child_rip+0x0/0x11