VM resources running in a cluster become unresponsive waiting on disk I/O and host node I/O to device also hanging in RHEL 6.2

Issue

• RHEL6.2 KVM hypervisor, RHEL6 guests with LVM volume as storage. Guests hang and hypervisor shows 100% utilization on LVM volumes.
• clvmd is hanging on cluster nodes with a hung_task backtrace in messages:

May 29 13:04:09 node1 kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
May 29 13:04:09 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 13:04:09 node1 kernel: clvmd         D 0000000000000000     0  4741      1 0x00000000
May 29 13:04:09 node1 kernel: ffff8817e1f89b18 0000000000000086 ffff8817e1f89ad8 ffffffffa00041fc
May 29 13:04:09 node1 kernel: ffff8817ebbed020 ffff880bead10600 0000000000000001 000000000000000c
May 29 13:04:09 node1 kernel: ffff8817e2f770f8 ffff8817e1f89fd8 000000000000f4e8 ffff8817e2f770f8
May 29 13:04:09 node1 kernel: Call Trace:
May 29 13:04:09 node1 kernel: [<ffffffffa00041fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
May 29 13:04:09 node1 kernel: [<ffffffff8109b849>] ? ktime_get_ts+0xa9/0xe0
May 29 13:04:09 node1 kernel: [<ffffffff814ed6e3>] io_schedule+0x73/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff811b1c9e>] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
May 29 13:04:09 node1 kernel: [<ffffffff811b218e>] __blockdev_direct_IO+0x5e/0xd0
May 29 13:04:09 node1 kernel: [<ffffffff811aea90>] ? blkdev_get_blocks+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff811af8f7>] blkdev_direct_IO+0x57/0x60
May 29 13:04:09 node1 kernel: [<ffffffff811aea90>] ? blkdev_get_blocks+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff81112a2b>] generic_file_aio_read+0x6bb/0x700
May 29 13:04:09 node1 kernel: [<ffffffff811b0360>] ? blkdev_open+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff8120d6af>] ? security_inode_permission+0x1f/0x30
May 29 13:04:09 node1 kernel: [<ffffffff8117665a>] do_sync_read+0xfa/0x140
May 29 13:04:09 node1 kernel: [<ffffffff811877da>] ? do_filp_open+0x6da/0xd60
May 29 13:04:09 node1 kernel: [<ffffffff81090c30>] ? autoremove_wake_function+0x0/0x40
May 29 13:04:09 node1 kernel: [<ffffffff811aeecc>] ? block_ioctl+0x3c/0x40
May 29 13:04:09 node1 kernel: [<ffffffff81189582>] ? vfs_ioctl+0x22/0xa0
May 29 13:04:09 node1 kernel: [<ffffffff81189724>] ? do_vfs_ioctl+0x84/0x580
May 29 13:04:09 node1 kernel: [<ffffffff8120c546>] ? security_file_permission+0x16/0x20
May 29 13:04:09 node1 kernel: [<ffffffff81177055>] vfs_read+0xb5/0x1a0
May 29 13:04:09 node1 kernel: [<ffffffff81177191>] sys_read+0x51/0x90
May 29 13:04:09 node1 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b