Why does RHEL 6.4 Hypervisor / Host using Boot from SAN hang on shutdown?

Solution In Progress - Updated -

Issue

  • SAN boot RHEV/RHEL hypervisors hangs on shutdown/reboot with the following error:
Shutting down vdsm daemon: 
vdsm watchdog stop[  OK  ]
vdsm stop[  OK  ]
Stopping puppet: [  OK  ]
Stopping rhsmcertd...[  OK  ]
Stopping libvirtd daemon: [  OK  ]
Stopping Red Hat Network Daemon: [  OK  ]
Sending stop signal sanlock (4658): [  OK  ]
Waiting for sanlock (4658) to stop:[  OK  ]
Sending stop signal wdmd (4566): [  OK  ]
Waiting for wdmd (4566) to stop:[  OK  ]
Stopping atd: [  OK  ]
Stopping Munin Node agents: [  OK  ]
Stopping ksmtuned: [  OK  ]
Stopping abrt daemon: [  OK  ]
Stopping sshd: [  OK  ]
Shutting down nrpe: [  OK  ]
Shutting down sm-client: [  OK  ]
Shutting down sendmail: [  OK  ]
Stopping snmpd: [  OK  ]
Stopping crond: [  OK  ]
Stopping acpi daemon: [  OK  ]
Stopping HAL daemon: [  OK  ]
Stopping nscd: [  OK  ]
Shutting down ntpd: [  OK  ]
Stopping block device availability: Deactivating block devices:
  [SKIP]: unmount of 360a980004433374b485d44414968557ap2 (dm-2) mounted on [SWAP]
  [SKIP]: unmount of 360a980004433374b485d44414968557ap3 (dm-3) mounted on /
  DM: deactivating part device 360a980004433374b485d44414968557ap1 (dm-1)
device-mapper: remove ioctl on 360a980004433374b485d44414968557ap1 failed: Device or resource busy
Command failed
INFO: task lvm:9641 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
lvm           D 0000000000000008     0  9641   9624 0x00000080
 ffff88206500bac8 0000000000000086 0000000000000000 ffff882066418538
 ffff882066418538 ffff882067336a00 0000000000000001 000000000000000c
 ffff882067d1bab8 ffff88206500bfd8 000000000000fb88 ffff882067d1bab8
Call Trace:
 [<ffffffff8150e513>] io_schedule+0x73/0xc0
 [<ffffffff811beb1e>] __blockdev_direct_IO_newtrunc+0x6de/0xb30
 [<ffffffff8119049f>] ? do_lookup+0x9f/0x230
 [<ffffffff811befce>] __blockdev_direct_IO+0x5e/0xd0
 [<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
 [<ffffffff811bc467>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
 [<ffffffff8111bbfb>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff812233e1>] ? avc_has_perm+0x71/0x90
 [<ffffffff811bce39>] ? __blkdev_get+0x1a9/0x3b0
 [<ffffffff811bb9b3>] blkdev_aio_read+0x53/0xc0
 [<ffffffff81180f5a>] do_sync_read+0xfa/0x140
 [<ffffffff81096cc0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811bb88c>] ? block_ioctl+0x3c/0x40
 [<ffffffff81195062>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff81228e0b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff8121bce6>] ? security_file_permission+0x16/0x20
 [<ffffffff81181845>] vfs_read+0xb5/0x1a0
 [<ffffffff81181981>] sys_read+0x51/0x90
 [<ffffffff810dc5b5>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
  • However, if the node is activated at all in RHEV, it will hang at shutdown, regardless of being in 'maintenance'

Environment

  • Red Hat Enterprise Linux 6.4
  • Red Hat Enterprise Virtualization 3.1 / 3.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In