RHEL6.9: NFS mounts hanging with rpciod blocked more than 120 seconds in xprt_put ... out_of_line_wait_on_bit_lock backtrace
Issue
- After updating to RHEL6.9 kernel, NFS mount is hung and rpciod process is blocked in a backtrace similar to the following
Jul 4 08:19:55 nfs-client kernel: INFO: task rpciod/8:3388 blocked for more than 120 seconds.
Jul 4 08:19:55 nfs-client kernel: Tainted: P -- ------------ 2.6.32-696.1.1.el6.x86_64 #1
Jul 4 08:19:55 nfs-client kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 4 08:19:55 nfs-client kernel: rpciod/8 D 0000000000000008 0 3388 2 0x00000000
Jul 4 08:19:55 nfs-client kernel: ffff8805fbf2fc20 0000000000000046 0000000000000000 ffffffff81ae7380
Jul 4 08:19:55 nfs-client kernel: ffff8805fbf2fbf0 ffffffff811ddca3 0000f297f4aa5664 ffff8805909d6a80
Jul 4 08:19:55 nfs-client kernel: 0000000000000000 000000010fdf363d ffff8806177bd068 ffff8805fbf2ffd8
Jul 4 08:19:55 nfs-client kernel: Call Trace:
Jul 4 08:19:55 nfs-client kernel: [<ffffffff811ddca3>] ? fsnotify+0x113/0x160
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4d0>] ? xprt_wait_bit+0x0/0x20 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4de>] xprt_wait_bit+0xe/0x20 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8154b78a>] __wait_on_bit_lock+0x5a/0xc0
Jul 4 08:19:55 nfs-client kernel: [<ffffffff811b6853>] ? d_delete+0xb3/0x2b0
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4d0>] ? xprt_wait_bit+0x0/0x20 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8154b868>] out_of_line_wait_on_bit_lock+0x78/0x90
Jul 4 08:19:55 nfs-client kernel: [<ffffffff810a68c0>] ? wake_bit_function+0x0/0x50
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdd45e>] xprt_put+0x5e/0xd0 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdc221>] rpc_free_client+0x91/0xe0 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdc2d8>] rpc_release_client+0x68/0xa0 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0bdc365>] rpc_task_release_client+0x55/0x70 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0be4fd4>] rpc_release_resources_task+0x34/0x40 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0be5ac4>] __rpc_execute+0x174/0x350 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0be5840>] ? rpc_wake_up_status+0x70/0x80 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0be5d40>] ? rpc_async_schedule+0x0/0x40 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffffa0be5d6a>] rpc_async_schedule+0x2a/0x40 [sunrpc]
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8109fb40>] worker_thread+0x170/0x2a0
Jul 4 08:19:55 nfs-client kernel: [<ffffffff810a6840>] ? autoremove_wake_function+0x0/0x40
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8109f9d0>] ? worker_thread+0x0/0x2a0
Jul 4 08:19:55 nfs-client kernel: [<ffffffff810a63ae>] kthread+0x9e/0xc0
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
Jul 4 08:19:55 nfs-client kernel: [<ffffffff810a6310>] ? kthread+0x0/0xc0
Jul 4 08:19:55 nfs-client kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20
Environment
- Red Hat Enterprise Linux 6.9 (NFS client)
- seen on kernel-2.6.32-696.1.1.el6
- NFSv4.0 with 'AUTH_GSS' (i.e. 'sec=krb5')
- seen with NetApp NFS server (OnTap version 8.1.4P2)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.