Why does RHEL 6.4 Hypervisor / Host using Boot from SAN hang on shutdown?
Issue
- SAN boot RHEV/RHEL hypervisors hangs on shutdown/reboot with the following error:
Shutting down vdsm daemon:
vdsm watchdog stop[ OK ]
vdsm stop[ OK ]
Stopping puppet: [ OK ]
Stopping rhsmcertd...[ OK ]
Stopping libvirtd daemon: [ OK ]
Stopping Red Hat Network Daemon: [ OK ]
Sending stop signal sanlock (4658): [ OK ]
Waiting for sanlock (4658) to stop:[ OK ]
Sending stop signal wdmd (4566): [ OK ]
Waiting for wdmd (4566) to stop:[ OK ]
Stopping atd: [ OK ]
Stopping Munin Node agents: [ OK ]
Stopping ksmtuned: [ OK ]
Stopping abrt daemon: [ OK ]
Stopping sshd: [ OK ]
Shutting down nrpe: [ OK ]
Shutting down sm-client: [ OK ]
Shutting down sendmail: [ OK ]
Stopping snmpd: [ OK ]
Stopping crond: [ OK ]
Stopping acpi daemon: [ OK ]
Stopping HAL daemon: [ OK ]
Stopping nscd: [ OK ]
Shutting down ntpd: [ OK ]
Stopping block device availability: Deactivating block devices:
[SKIP]: unmount of 360a980004433374b485d44414968557ap2 (dm-2) mounted on [SWAP]
[SKIP]: unmount of 360a980004433374b485d44414968557ap3 (dm-3) mounted on /
DM: deactivating part device 360a980004433374b485d44414968557ap1 (dm-1)
device-mapper: remove ioctl on 360a980004433374b485d44414968557ap1 failed: Device or resource busy
Command failed
INFO: task lvm:9641 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
lvm D 0000000000000008 0 9641 9624 0x00000080
ffff88206500bac8 0000000000000086 0000000000000000 ffff882066418538
ffff882066418538 ffff882067336a00 0000000000000001 000000000000000c
ffff882067d1bab8 ffff88206500bfd8 000000000000fb88 ffff882067d1bab8
Call Trace:
[<ffffffff8150e513>] io_schedule+0x73/0xc0
[<ffffffff811beb1e>] __blockdev_direct_IO_newtrunc+0x6de/0xb30
[<ffffffff8119049f>] ? do_lookup+0x9f/0x230
[<ffffffff811befce>] __blockdev_direct_IO+0x5e/0xd0
[<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
[<ffffffff811bc467>] blkdev_direct_IO+0x57/0x60
[<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
[<ffffffff8111bbfb>] generic_file_aio_read+0x6bb/0x700
[<ffffffff812233e1>] ? avc_has_perm+0x71/0x90
[<ffffffff811bce39>] ? __blkdev_get+0x1a9/0x3b0
[<ffffffff811bb9b3>] blkdev_aio_read+0x53/0xc0
[<ffffffff81180f5a>] do_sync_read+0xfa/0x140
[<ffffffff81096cc0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff811bb88c>] ? block_ioctl+0x3c/0x40
[<ffffffff81195062>] ? vfs_ioctl+0x22/0xa0
[<ffffffff81228e0b>] ? selinux_file_permission+0xfb/0x150
[<ffffffff8121bce6>] ? security_file_permission+0x16/0x20
[<ffffffff81181845>] vfs_read+0xb5/0x1a0
[<ffffffff81181981>] sys_read+0x51/0x90
[<ffffffff810dc5b5>] ? __audit_syscall_exit+0x265/0x290
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
- However, if the node is activated at all in RHEV, it will hang at shutdown, regardless of being in 'maintenance'
Environment
- Red Hat Enterprise Linux 6.4
- Red Hat Enterprise Virtualization 3.1 / 3.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.