Why does RHEL 6.4 Hypervisor / Host using Boot from SAN hang on shutdown?

Solution In Progress - Updated -

Issue

  • SAN boot RHEV/RHEL hypervisors hangs on shutdown/reboot with the following error:
Shutting down vdsm daemon: 
vdsm watchdog stop[  OK  ]
vdsm stop[  OK  ]
Stopping puppet: [  OK  ]
Stopping rhsmcertd...[  OK  ]
Stopping libvirtd daemon: [  OK  ]
Stopping Red Hat Network Daemon: [  OK  ]
Sending stop signal sanlock (4658): [  OK  ]
Waiting for sanlock (4658) to stop:[  OK  ]
Sending stop signal wdmd (4566): [  OK  ]
Waiting for wdmd (4566) to stop:[  OK  ]
Stopping atd: [  OK  ]
Stopping Munin Node agents: [  OK  ]
Stopping ksmtuned: [  OK  ]
Stopping abrt daemon: [  OK  ]
Stopping sshd: [  OK  ]
Shutting down nrpe: [  OK  ]
Shutting down sm-client: [  OK  ]
Shutting down sendmail: [  OK  ]
Stopping snmpd: [  OK  ]
Stopping crond: [  OK  ]
Stopping acpi daemon: [  OK  ]
Stopping HAL daemon: [  OK  ]
Stopping nscd: [  OK  ]
Shutting down ntpd: [  OK  ]
Stopping block device availability: Deactivating block devices:
  [SKIP]: unmount of 360a980004433374b485d44414968557ap2 (dm-2) mounted on [SWAP]
  [SKIP]: unmount of 360a980004433374b485d44414968557ap3 (dm-3) mounted on /
  DM: deactivating part device 360a980004433374b485d44414968557ap1 (dm-1)
device-mapper: remove ioctl on 360a980004433374b485d44414968557ap1 failed: Device or resource busy
Command failed
INFO: task lvm:9641 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
lvm           D 0000000000000008     0  9641   9624 0x00000080
 ffff88206500bac8 0000000000000086 0000000000000000 ffff882066418538
 ffff882066418538 ffff882067336a00 0000000000000001 000000000000000c
 ffff882067d1bab8 ffff88206500bfd8 000000000000fb88 ffff882067d1bab8
Call Trace:
 [<ffffffff8150e513>] io_schedule+0x73/0xc0
 [<ffffffff811beb1e>] __blockdev_direct_IO_newtrunc+0x6de/0xb30
 [<ffffffff8119049f>] ? do_lookup+0x9f/0x230
 [<ffffffff811befce>] __blockdev_direct_IO+0x5e/0xd0
 [<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
 [<ffffffff811bc467>] blkdev_direct_IO+0x57/0x60
 [<ffffffff811bb3a0>] ? blkdev_get_blocks+0x0/0xc0
 [<ffffffff8111bbfb>] generic_file_aio_read+0x6bb/0x700
 [<ffffffff812233e1>] ? avc_has_perm+0x71/0x90
 [<ffffffff811bce39>] ? __blkdev_get+0x1a9/0x3b0
 [<ffffffff811bb9b3>] blkdev_aio_read+0x53/0xc0
 [<ffffffff81180f5a>] do_sync_read+0xfa/0x140
 [<ffffffff81096cc0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff811bb88c>] ? block_ioctl+0x3c/0x40
 [<ffffffff81195062>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff81228e0b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff8121bce6>] ? security_file_permission+0x16/0x20
 [<ffffffff81181845>] vfs_read+0xb5/0x1a0
 [<ffffffff81181981>] sys_read+0x51/0x90
 [<ffffffff810dc5b5>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
  • However, if the node is activated at all in RHEV, it will hang at shutdown, regardless of being in 'maintenance'

Environment

  • Red Hat Enterprise Linux 6.4
  • Red Hat Enterprise Virtualization 3.1 / 3.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content