VM resources running in a cluster become unresponsive waiting on disk I/O and host node I/O to device also hanging in RHEL 6.2
Issue
- RHEL6.2 KVM hypervisor, RHEL6 guests with LVM volume as storage. Guests hang and hypervisor shows 100% utilization on LVM volumes.
- clvmd is hanging on cluster nodes with a hung_task backtrace in messages:
May 29 13:04:09 node1 kernel: INFO: task clvmd:4741 blocked for more than 120 seconds.
May 29 13:04:09 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 13:04:09 node1 kernel: clvmd D 0000000000000000 0 4741 1 0x00000000
May 29 13:04:09 node1 kernel: ffff8817e1f89b18 0000000000000086 ffff8817e1f89ad8 ffffffffa00041fc
May 29 13:04:09 node1 kernel: ffff8817ebbed020 ffff880bead10600 0000000000000001 000000000000000c
May 29 13:04:09 node1 kernel: ffff8817e2f770f8 ffff8817e1f89fd8 000000000000f4e8 ffff8817e2f770f8
May 29 13:04:09 node1 kernel: Call Trace:
May 29 13:04:09 node1 kernel: [<ffffffffa00041fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
May 29 13:04:09 node1 kernel: [<ffffffff8109b849>] ? ktime_get_ts+0xa9/0xe0
May 29 13:04:09 node1 kernel: [<ffffffff814ed6e3>] io_schedule+0x73/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff811b1c9e>] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
May 29 13:04:09 node1 kernel: [<ffffffff811b218e>] __blockdev_direct_IO+0x5e/0xd0
May 29 13:04:09 node1 kernel: [<ffffffff811aea90>] ? blkdev_get_blocks+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff811af8f7>] blkdev_direct_IO+0x57/0x60
May 29 13:04:09 node1 kernel: [<ffffffff811aea90>] ? blkdev_get_blocks+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff81112a2b>] generic_file_aio_read+0x6bb/0x700
May 29 13:04:09 node1 kernel: [<ffffffff811b0360>] ? blkdev_open+0x0/0xc0
May 29 13:04:09 node1 kernel: [<ffffffff8120d6af>] ? security_inode_permission+0x1f/0x30
May 29 13:04:09 node1 kernel: [<ffffffff8117665a>] do_sync_read+0xfa/0x140
May 29 13:04:09 node1 kernel: [<ffffffff811877da>] ? do_filp_open+0x6da/0xd60
May 29 13:04:09 node1 kernel: [<ffffffff81090c30>] ? autoremove_wake_function+0x0/0x40
May 29 13:04:09 node1 kernel: [<ffffffff811aeecc>] ? block_ioctl+0x3c/0x40
May 29 13:04:09 node1 kernel: [<ffffffff81189582>] ? vfs_ioctl+0x22/0xa0
May 29 13:04:09 node1 kernel: [<ffffffff81189724>] ? do_vfs_ioctl+0x84/0x580
May 29 13:04:09 node1 kernel: [<ffffffff8120c546>] ? security_file_permission+0x16/0x20
May 29 13:04:09 node1 kernel: [<ffffffff81177055>] vfs_read+0xb5/0x1a0
May 29 13:04:09 node1 kernel: [<ffffffff81177191>] sys_read+0x51/0x90
May 29 13:04:09 node1 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux (RHEL) 6 Update 2 with the High Availability Add On
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
