RHEL6: many threads including xfssyncd and flush hang xlog_grant_head_wait

Issue

• High system load with the load due to D-state (uninterruptible sleep) processes
• Server hung with many threads, including xfssyncd and flush, hung in xlog_grant_head_wait with backtraces similar to:

INFO: task xfssyncd/dm-26:3207 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfssyncd/dm-2 D 0000000000000011     0  3207      2 0x00000080
 ffff881064e8fcd0 0000000000000046 0000000000000000 0000000000000002
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 ffff88106684b098 ffff881064e8ffd8 000000000000fb88 ffff88106684b098
Call Trace:
 [<ffffffffa02f7b99>] xlog_grant_head_wait+0xa9/0x1a0 [xfs]
 [<ffffffffa02f7d5e>] xlog_grant_head_check+0xce/0xf0 [xfs]
 [<ffffffffa02f7e50>] xfs_log_reserve+0xd0/0x230 [xfs]
 [<ffffffffa03059d7>] xfs_trans_reserve+0x227/0x240 [xfs]
 [<ffffffffa030ce3a>] ? kmem_zone_zalloc+0x3a/0x50 [xfs]
 [<ffffffffa02e9273>] xfs_fs_log_dummy+0x43/0x90 [xfs]
 [<ffffffffa02f6f54>] ? xfs_log_need_covered+0x94/0xd0 [xfs]
 [<ffffffffa031b8d0>] xfs_sync_worker+0x40/0xa0 [xfs]
 [<ffffffffa031b7fe>] xfssyncd+0x17e/0x210 [xfs]
 [<ffffffffa031b680>] ? xfssyncd+0x0/0x210 [xfs]
 [<ffffffff81096936>] kthread+0x96/0xa0
 [<ffffffff8100c0ca>] child_rip+0xa/0x20
 [<ffffffff810968a0>] ? kthread+0x0/0xa0
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20