RHEL6.9: NFS mounts hanging with rpciod blocked more than 120 seconds in xprt_put ... out_of_line_wait_on_bit_lock backtrace

Solution In Progress - Updated -

Issue

  • After updating to RHEL6.9 kernel, NFS mount is hung and rpciod process is blocked in a backtrace similar to the following
Jul  4 08:19:55 nfs-client kernel: INFO: task rpciod/8:3388 blocked for more than 120 seconds.
Jul  4 08:19:55 nfs-client kernel: Tainted: P           -- ------------    2.6.32-696.1.1.el6.x86_64 #1
Jul  4 08:19:55 nfs-client kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  4 08:19:55 nfs-client kernel: rpciod/8      D 0000000000000008     0  3388      2 0x00000000
Jul  4 08:19:55 nfs-client kernel: ffff8805fbf2fc20 0000000000000046 0000000000000000 ffffffff81ae7380
Jul  4 08:19:55 nfs-client kernel: ffff8805fbf2fbf0 ffffffff811ddca3 0000f297f4aa5664 ffff8805909d6a80
Jul  4 08:19:55 nfs-client kernel: 0000000000000000 000000010fdf363d ffff8806177bd068 ffff8805fbf2ffd8
Jul  4 08:19:55 nfs-client kernel: Call Trace:
Jul  4 08:19:55 nfs-client kernel: [<ffffffff811ddca3>] ? fsnotify+0x113/0x160
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4d0>] ? xprt_wait_bit+0x0/0x20 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4de>] xprt_wait_bit+0xe/0x20 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8154b78a>] __wait_on_bit_lock+0x5a/0xc0
Jul  4 08:19:55 nfs-client kernel: [<ffffffff811b6853>] ? d_delete+0xb3/0x2b0
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdd4d0>] ? xprt_wait_bit+0x0/0x20 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8154b868>] out_of_line_wait_on_bit_lock+0x78/0x90
Jul  4 08:19:55 nfs-client kernel: [<ffffffff810a68c0>] ? wake_bit_function+0x0/0x50
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdd45e>] xprt_put+0x5e/0xd0 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdc221>] rpc_free_client+0x91/0xe0 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdc2d8>] rpc_release_client+0x68/0xa0 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0bdc365>] rpc_task_release_client+0x55/0x70 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0be4fd4>] rpc_release_resources_task+0x34/0x40 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0be5ac4>] __rpc_execute+0x174/0x350 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0be5840>] ? rpc_wake_up_status+0x70/0x80 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0be5d40>] ? rpc_async_schedule+0x0/0x40 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffffa0be5d6a>] rpc_async_schedule+0x2a/0x40 [sunrpc]
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8109fb40>] worker_thread+0x170/0x2a0
Jul  4 08:19:55 nfs-client kernel: [<ffffffff810a6840>] ? autoremove_wake_function+0x0/0x40
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8109f9d0>] ? worker_thread+0x0/0x2a0
Jul  4 08:19:55 nfs-client kernel: [<ffffffff810a63ae>] kthread+0x9e/0xc0
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
Jul  4 08:19:55 nfs-client kernel: [<ffffffff810a6310>] ? kthread+0x0/0xc0
Jul  4 08:19:55 nfs-client kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20

Environment

  • Red Hat Enterprise Linux 6.9 (NFS client)
    • seen on kernel-2.6.32-696.1.1.el6
  • NFSv4.0 with 'AUTH_GSS' (i.e. 'sec=krb5')
  • seen with NetApp NFS server (OnTap version 8.1.4P2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content