GlusterFS Native client mounts hang a RHEL-6 server and print stacktraces in the logs

Issue

• We are seeing GlusterFS mounts hanging. Only rebooting the servers cleared the problem. These stacktraces are repeatedly listed in /var/log/messages:

INFO: task glusterfs:32632 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfs     D ffff88043fc23280     0 32632      1 0x00000080
ffff88042b835548 0000000000000086 0000000000000000 0000000000000297
ffff88042b8354b8 ffffea0003b733b0 000000000000000e 000000013376dabb
ffff88042bf1db18 ffff88042b835fd8 0000000000010518 ffff88042bf1db18
Call Trace:
[<ffffffff8110c060>] ? sync_page+0x0/0x50
[<ffffffff814c8a23>] io_schedule+0x73/0xc0
[<ffffffff8110c09d>] sync_page+0x3d/0x50
[<ffffffff814c914a>] __wait_on_bit_lock+0x5a/0xc0
[<ffffffff8110c037>] __lock_page+0x67/0x70
[<ffffffff81091ce0>] ? wake_bit_function+0x0/0x50
[<ffffffff81122461>] ? lru_cache_add_lru+0x21/0x40
[<ffffffff8115b730>] lock_page+0x30/0x40
[<ffffffff8115bdad>] migrate_pages+0x59d/0x5d0
[<ffffffff811223b7>] ? ____pagevec_lru_add+0x167/0x180
[<ffffffff81152470>] ? compaction_alloc+0x0/0x370
[<ffffffff81151f1c>] compact_zone+0x4ac/0x5e0
[<ffffffff8111cd1c>] ? get_page_from_freelist+0x15c/0x820
[<ffffffff811522ce>] compact_zone_order+0x7e/0xb0
[<ffffffff81152409>] try_to_compact_pages+0x109/0x170
[<ffffffff8111e62c>] __alloc_pages_nodemask+0x55c/0x810
[<ffffffff81150374>] alloc_pages_vma+0x84/0x110
[<ffffffff8113ef50>] ? anon_vma_prepare+0x30/0x160
[<ffffffff811673b5>] do_huge_pmd_anonymous_page+0x135/0x360
[<ffffffff81136455>] handle_mm_fault+0x245/0x2b0
[<ffffffff811365e9>] __get_user_pages+0x129/0x420
[<ffffffff81131456>] ? follow_page+0x3e6/0x460
[<ffffffff81136979>] get_user_pages+0x49/0x50
[<ffffffffa03bd44a>] fuse_copy_fill+0xca/0x1f0 [fuse]
[<ffffffffa03bd796>] fuse_copy_args+0x1b6/0x350 [fuse]
[<ffffffffa03be524>] fuse_dev_read+0x284/0x310 [fuse]
[<ffffffffa03be2a0>] ? fuse_dev_read+0x0/0x310 [fuse]
[<ffffffff8116c3db>] do_sync_readv_writev+0xfb/0x140
[<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8113bb8c>] ? mmap_region+0x26c/0x5a0
[<ffffffff811ff3b6>] ? security_file_permission+0x16/0x20
[<ffffffff8116d49f>] do_readv_writev+0xcf/0x1f0
[<ffffffff8116d7d3>] vfs_readv+0x43/0x60
[<ffffffff8116d901>] sys_readv+0x51/0xb0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b

• And also there is the rare occasion of stacktraces from khugepaged:

INFO: task khugepaged:52 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
khugepaged    D ffff88043fc23080     0    52      2 0x00000000
ffff880429ee1c80 0000000000000046 0000000000000000 0000000000016980
0000000000000000 ffff880428739740 ffffffffffffffff 0000000133840e92
ffff880429e66638 ffff880429ee1fd8 0000000000010518 ffff880429e66638
Call Trace:
[<ffffffff814ca6b5>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ca813>] rwsem_down_write_failed+0x23/0x30
[<ffffffff81264253>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814c9d12>] ? down_write+0x32/0x40
[<ffffffff81165002>] khugepaged+0x742/0x1190
[<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811648c0>] ? khugepaged+0x0/0x1190
[<ffffffff81091936>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff810918a0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20