GlusterFS Native client mounts hang a RHEL-6 server and print stacktraces in the logs
Issue
- We are seeing GlusterFS mounts hanging. Only rebooting the servers cleared the problem. These stacktraces are repeatedly listed in
/var/log/messages:
INFO: task glusterfs:32632 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
glusterfs D ffff88043fc23280 0 32632 1 0x00000080
ffff88042b835548 0000000000000086 0000000000000000 0000000000000297
ffff88042b8354b8 ffffea0003b733b0 000000000000000e 000000013376dabb
ffff88042bf1db18 ffff88042b835fd8 0000000000010518 ffff88042bf1db18
Call Trace:
[<ffffffff8110c060>] ? sync_page+0x0/0x50
[<ffffffff814c8a23>] io_schedule+0x73/0xc0
[<ffffffff8110c09d>] sync_page+0x3d/0x50
[<ffffffff814c914a>] __wait_on_bit_lock+0x5a/0xc0
[<ffffffff8110c037>] __lock_page+0x67/0x70
[<ffffffff81091ce0>] ? wake_bit_function+0x0/0x50
[<ffffffff81122461>] ? lru_cache_add_lru+0x21/0x40
[<ffffffff8115b730>] lock_page+0x30/0x40
[<ffffffff8115bdad>] migrate_pages+0x59d/0x5d0
[<ffffffff811223b7>] ? ____pagevec_lru_add+0x167/0x180
[<ffffffff81152470>] ? compaction_alloc+0x0/0x370
[<ffffffff81151f1c>] compact_zone+0x4ac/0x5e0
[<ffffffff8111cd1c>] ? get_page_from_freelist+0x15c/0x820
[<ffffffff811522ce>] compact_zone_order+0x7e/0xb0
[<ffffffff81152409>] try_to_compact_pages+0x109/0x170
[<ffffffff8111e62c>] __alloc_pages_nodemask+0x55c/0x810
[<ffffffff81150374>] alloc_pages_vma+0x84/0x110
[<ffffffff8113ef50>] ? anon_vma_prepare+0x30/0x160
[<ffffffff811673b5>] do_huge_pmd_anonymous_page+0x135/0x360
[<ffffffff81136455>] handle_mm_fault+0x245/0x2b0
[<ffffffff811365e9>] __get_user_pages+0x129/0x420
[<ffffffff81131456>] ? follow_page+0x3e6/0x460
[<ffffffff81136979>] get_user_pages+0x49/0x50
[<ffffffffa03bd44a>] fuse_copy_fill+0xca/0x1f0 [fuse]
[<ffffffffa03bd796>] fuse_copy_args+0x1b6/0x350 [fuse]
[<ffffffffa03be524>] fuse_dev_read+0x284/0x310 [fuse]
[<ffffffffa03be2a0>] ? fuse_dev_read+0x0/0x310 [fuse]
[<ffffffff8116c3db>] do_sync_readv_writev+0xfb/0x140
[<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8113bb8c>] ? mmap_region+0x26c/0x5a0
[<ffffffff811ff3b6>] ? security_file_permission+0x16/0x20
[<ffffffff8116d49f>] do_readv_writev+0xcf/0x1f0
[<ffffffff8116d7d3>] vfs_readv+0x43/0x60
[<ffffffff8116d901>] sys_readv+0x51/0xb0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
- And also there is the rare occasion of stacktraces from
khugepaged:
INFO: task khugepaged:52 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
khugepaged D ffff88043fc23080 0 52 2 0x00000000
ffff880429ee1c80 0000000000000046 0000000000000000 0000000000016980
0000000000000000 ffff880428739740 ffffffffffffffff 0000000133840e92
ffff880429e66638 ffff880429ee1fd8 0000000000010518 ffff880429e66638
Call Trace:
[<ffffffff814ca6b5>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ca813>] rwsem_down_write_failed+0x23/0x30
[<ffffffff81264253>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814c9d12>] ? down_write+0x32/0x40
[<ffffffff81165002>] khugepaged+0x742/0x1190
[<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811648c0>] ? khugepaged+0x0/0x1190
[<ffffffff81091936>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff810918a0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20
Environment
- Red Hat Enterprise Linux 6
- GlusterFS Native client
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
