RHEL7: One program writing to NFS hangs under nfs_wait_bit_uninterruptible while the rest of NFS remains responsive.

Solution Verified - Updated -

Issue

  • Defunct process is stuck in in nfs_wait_bit_interruptible
  • Process writing to an NFS file becomes stuck while other processes access NFS without issue
  • RHEL 7.2 system is getting hung while taking backup on NFS
Apr 25 22:44:30 localhost kernel: INFO: task kernel:22442 blocked for more than 120 seconds.
Apr 25 22:44:30 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 25 22:44:30 localhost kernel: kernel          D ffff8823c49d7b30     0 22442   5299 0x00000080
Apr 25 22:44:30 localhost kernel: ffff8823c49d79d0 0000000000000086 ffff880a04789700 ffff8823c49d7fd8
Apr 25 22:44:30 localhost kernel: ffff8823c49d7fd8 ffff8823c49d7fd8 ffff880a04789700 ffff88283fc14780
Apr 25 22:44:30 localhost kernel: 0000000000000000 7fffffffffffffff ffffffffa0598bc0 ffff8823c49d7b30
Apr 25 22:44:30 localhost kernel: Call Trace:
Apr 25 22:44:30 localhost kernel: [<ffffffffa0598bc0>] ? nfs_pgio_header_free+0x20/0x20 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff8163a879>] schedule+0x29/0x70
Apr 25 22:44:30 localhost kernel: [<ffffffff81638569>] schedule_timeout+0x209/0x2d0
Apr 25 22:44:30 localhost kernel: [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
Apr 25 22:44:30 localhost kernel: [<ffffffff8101c829>] ? read_tsc+0x9/0x10
Apr 25 22:44:30 localhost kernel: [<ffffffffa0598bc0>] ? nfs_pgio_header_free+0x20/0x20 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff81639eae>] io_schedule_timeout+0xae/0x130
Apr 25 22:44:30 localhost kernel: [<ffffffff81639f48>] io_schedule+0x18/0x20
Apr 25 22:44:30 localhost kernel: [<ffffffffa0598bce>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff816386f0>] __wait_on_bit+0x60/0x90
Apr 25 22:44:30 localhost kernel: [<ffffffffa0598bc0>] ? nfs_pgio_header_free+0x20/0x20 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff816387a7>] out_of_line_wait_on_bit+0x87/0xb0
Apr 25 22:44:30 localhost kernel: [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
Apr 25 22:44:30 localhost kernel: [<ffffffffa059a593>] nfs_wait_on_request+0x33/0x40 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffffa059f161>] nfs_updatepage+0x121/0x8a0 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffffa058efa1>] nfs_write_end+0x121/0x350 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff81169d74>] generic_file_buffered_write+0x184/0x290
Apr 25 22:44:30 localhost kernel: [<ffffffff8116b235>] __generic_file_aio_write+0x1d5/0x3e0
Apr 25 22:44:30 localhost kernel: [<ffffffff8116b49d>] generic_file_aio_write+0x5d/0xc0
Apr 25 22:44:30 localhost kernel: [<ffffffffa058e03b>] nfs_file_write+0xbb/0x1d0 [nfs]
Apr 25 22:44:30 localhost kernel: [<ffffffff811ddcbd>] do_sync_write+0x8d/0xd0
Apr 25 22:44:30 localhost kernel: [<ffffffff811de4dd>] vfs_write+0xbd/0x1e0
Apr 25 22:44:30 localhost kernel: [<ffffffff811def7f>] SyS_write+0x7f/0xe0
Apr 25 22:44:30 localhost kernel: [<ffffffff816458c9>] system_call_fastpath+0x16/0x1

Environment

  • Red Hat Enterprise Linux 7 (NFS client)
    • kernel earlier than kernel-3.10.0-327.42.1.el7
  • NFS

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content