RHEL5: nfs server hangs or hung_task panic, while accessing files on pmfs filesystem

Solution Unverified - Updated -

Issue

  • nfsd threads stuck in backtraces waiting on uvfs_make_request, similar to
PID: 17735  TASK: ffff8103ef8c67b0  CPU: 1   COMMAND: "nfsd"
 #0 [ffff81010fe0bb00] schedule at ffffffff80062fa0
 #1 [ffff81010fe0bbd8] uvfs_make_request at ffffffff8874d4ea [pmfs]
 #2 [ffff81010fe0bc38] uvfs_lookup_by_name at ffffffff8874c0d5 [pmfs]
 #3 [ffff81010fe0bc78] uvfs_dentry_revalidate at ffffffff8874c1c6 [pmfs]
 #4 [ffff81010fe0bd08] __lookup_hash at ffffffff800371ce
 #5 [ffff81010fe0bd48] lookup_one_len at ffffffff800edb09
 #6 [ffff81010fe0bd68] nfsd_lookup_dentry at ffffffff88780b1b [nfsd]
 #7 [ffff81010fe0bdd8] nfsd_lookup at ffffffff88780be2 [nfsd]
 #8 [ffff81010fe0be18] nfsd_proc_lookup at ffffffff8877dc39 [nfsd]
 #9 [ffff81010fe0be48] nfsd_dispatch at ffffffff8877c1db [nfsd]
#10 [ffff81010fe0be78] svc_process at ffffffff885db83e [sunrpc]
#11 [ffff81010fe0bee8] nfsd at ffffffff8877c770 [nfsd]
#12 [ffff81010fe0bf48] kernel_thread at ffffffff8005dfc1
  • a hung_task backtrace and/or panic may be triggered eventually, due to another nfsd thread waiting on an inode mutex, held by one of the processes waiting on uvfs_make_request
INFO: task nfsd:17740 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nfsd          D ffff81000900c9a0     0 17740      1         17739 17733 (L-TLB)
 ffff810396dd9d00 0000000000000046 ffff810396dd9d60 ffffffff80077b6e
 0000000000000008 000000000000000a ffff8103743c2040 ffff81010ef4a0c0
 000292afccc20ccd 000000000001307e ffff8103743c2228 000000018874c958
Call Trace:
 [<ffffffff80077b6e>] smp_send_reschedule+0x4e/0x53
 [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
 [<ffffffff80063aff>] mutex_lock+0x13/0x1d
 [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
 [<ffffffff88780aab>] :nfsd:nfsd_lookup_dentry+0x32b/0x440
 [<ffffffff885de684>] :sunrpc:ip_map_match+0x19/0x30
 [<ffffffff88780be2>] :nfsd:nfsd_lookup+0x22/0xb0
 [<ffffffff885de76e>] :sunrpc:ip_map_lookup+0xbc/0xc3
 [<ffffffff8877dc39>] :nfsd:nfsd_proc_lookup+0x7c/0x98
 [<ffffffff8877c1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
 [<ffffffff885db83e>] :sunrpc:svc_process+0x3f8/0x6bf
 [<ffffffff80064624>] __down_read+0x12/0x92
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8877c770>] :nfsd:nfsd+0x1a5/0x2c9
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8877c5cb>] :nfsd:nfsd+0x0/0x2c9
 [<ffffffff8005dfb7>] child_rip+0x0/0x11

Environment

  • Red Hat Enterprise Linux 5 (NFS server)
    • see on kernel 2.6.18-371.11.1.el5
  • NFS server export on Persistent Memory File System (pmfs)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.