Why does RHEL 6.4 Hypervisor / Host using Boot from SAN hang on shutdown?
Issue
- SAN boot RHEV/RHEL hypervisors hangs on shutdown/reboot with the following error:
Shutting down vdsm daemon:
vdsm watchdog stop[ OK ]
vdsm stop[ OK ]
Stopping puppet: [ OK ]
Stopping rhsmcertd...[ OK ]
Stopping libvirtd daemon: [ OK ]
Stopping Red Hat Network Daemon: [ OK ]
Sending stop signal sanlock (4658): [ OK ]
Waiting for sanlock (4658) to stop:[ OK ]
Sending stop signal wdmd (4566): [ OK ]
Waiting for wdmd (4566) to stop:[ OK ]
Stopping atd: [ OK ]
Stopping Munin Node agents: [ OK ]
Stopping ksmtuned: [ OK ]
Stopping abrt daemon: [ OK ]
Stopping sshd: [ OK ]
Shutting down nrpe: [ OK ]
Shutting down sm-client: [ OK ]
Shutting down sendmail: [ OK ]
Stopping snmpd: [ OK ]
Stopping crond: [ OK ]
Stopping acpi daemon: [ OK ]
Stopping HAL daemon: [ OK ]
Stopping nscd: [ OK ]
Shutting down ntpd: [ OK ]
Stopping block device availability: Deactivating block devices:
[SKIP]: unmount of 360a980004433374b485d44414968557ap2 (dm-2) mounted on [SWAP]
[SKIP]: unmount of 360a980004433374b485d44414968557ap3 (dm-3) mounted on /
DM: deactivating part device 360a980004433374b485d44414968557ap1 (dm-1)
device-mapper: remove ioctl on 360a980004433374b485d44414968557ap1 failed: Device or resource busy
Command failed
INFO: task lvm:9641 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
lvm D 0000000000000008 0 9641 9624 0x00000080
ffff88206500bac8 0000000000000086 0000000000000000 ffff882066418538
ffff882066418538 ffff882067336a00 0000000000000001 000000000000000c
ffff882067d1bab8 ffff88206500bfd8 000000000000fb88 ffff882067d1bab8
Call Trace:
[<ffffffff8150e513>] io_schedule+0x73/0xc0
[<ffffffff811beb1e>] __blockdev_direct_IO_newtrunc+0x6de/0xb30
[<ffffffff8119049f>] ? do_lookup+0x9f/0x230
[<ffffffff811befce>] __blockdev_direct_IO+0x5e/0xd0
[<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
[<ffffffff811bc467>] blkdev_direct_IO+0x57/0x60
[<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
[<ffffffff8111bbfb>] generic_file_aio_read+0x6bb/0x700
[<ffffffff812233e1>] ? avc_has_perm+0x71/0x90
[<ffffffff811bce39>] ? __blkdev_get+0x1a9/0x3b0
[<ffffffff811bb9b3>] blkdev_aio_read+0x53/0xc0
[<ffffffff81180f5a>] do_sync_read+0xfa/0x140
[<ffffffff81096cc0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811bb88c>] ? block_ioctl+0x3c/0x40
[<ffffffff81195062>] ? vfs_ioctl+0x22/0xa0
[<ffffffff81228e0b>] ? selinux_file_permission+0xfb/0x150
[<ffffffff8121bce6>] ? security_file_permission+0x16/0x20
[<ffffffff81181845>] vfs_read+0xb5/0x1a0
[<ffffffff81181981>] sys_read+0x51/0x90
[<ffffffff810dc5b5>] ? __audit_syscall_exit+0x265/0x290
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
- However, if the node is activated at all in RHEV, it will hang at shutdown, regardless of being in 'maintenance'
Environment
- Red Hat Enterprise Linux 6.4
- Red Hat Enterprise Virtualization 3.1 / 3.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
