If volume(s) in illegal state, VM (Virtual Machine) fails to start

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.6.6
  • Red Hat Virtualization 4.x

Issue

  • VM (Virtual Machine) fails to start
  • Error message observed:
VM GUEST was started by admin@internal (Host: HOST.example.com).

VM GUEST is down with error. Exit message: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'68260611-aa4a-4e79-b093-abcdefghijkl', 'apparentsize': '5368709120', u'imageID': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', u'specParams': {}, u'readonly': u'false', u'domainID': u'97724cca-4600-4508-b234-abcdefghijkl', 'reqsize': '0', u'deviceId': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', 'truesize': '5368709120', u'poolID': u'da7792ba-7398-4577-9a50-bb8b6cfd0fe7', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'}.

Failed to run VM GUEST on Host HOST.example.com.

Resolution

  • The solution for this is to review logs to determine why the image was set to illegal status. Please contact Red Hat Technical Support and open a case.

Root Cause

  • An image/volume for the VM has illegal status.
Thread-820076::ERROR::2016-06-22 15:30:26,673::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot prepare illegal volume: ('68260611-aa4a-4e79-b093-abcdefghijkl',)", 'code': 227}}

Thread-820076::ERROR::2016-06-22 15:30:26,673::vm::759::virt.vm::(_startUnderlyingVm) vmId=`25c88009-5927-45dc-a434-abcdefghijkl`::The vm start process failed
  • This is by design. The fix to BZ BZ 1332038, i.e. vdsm-4.17.28-0 and later versions, now prevents a VM from starting if a volume is in an illegal state. This is to prevent potential data corruption.

Diagnostic Steps

  • VDSM log extract from the SPM host
Thread-820076::ERROR::2016-06-22 15:30:26,670::task::866::Storage.TaskManager.Task::(_setError) Task=`23fad322-060e-4ebd-bf95-e771393f1697`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 49, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3227, in prepareImage
    raise se.prepareIllegalVolumeError(volUUID)
prepareIllegalVolumeError: Cannot prepare illegal volume: ('68260611-aa4a-4e79-b093-abcdefghijkl',)

Thread-820076::ERROR::2016-06-22 15:30:26,673::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot prepare illegal volume: ('68260611-aa4a-4e79-b093-abcdefghijkl',)", 'code': 227}}
Thread-820076::ERROR::2016-06-22 15:30:26,673::vm::759::virt.vm::(_startUnderlyingVm) vmId=`25c88009-5927-45dc-a434-abcdefghijkl`::The vm start process failed

Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/virt/vm.py", line 1859, in _run
    self.preparePaths(dev_spec_map[hwclass.DISK])
  File "/usr/share/vdsm/virt/vm.py", line 780, in preparePaths
    drive['path'] = self.cif.prepareVolumePath(drive, self.id)
  File "/usr/share/vdsm/clientIF.py", line 317, in prepareVolumePath
    raise vm.VolumeError(drive)
VolumeError: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'68260611-aa4a-4e79-b093-abcdefghijkl', 'apparentsize': '5368709120', u'imageID': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', u'specParams': {}, u'readonly': u'false', u'domainID': u'97724cca-4600-4508-b234-abcdefghijkl', 'reqsize': '0', u'deviceId': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', 'truesize': '5368709120', u'poolID': u'da7792ba-7398-4577-9a50-bb8b6cfd0fe7', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'}
  • Engine log extract:
2016-06-22 09:42:51,086 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [c955d78] Correlation ID: c955d78, Job ID: 998842cc-ac12-4683-937e-2b7f3e31cc25, Call Stack: null, Custom Event ID: -1, Message: VM GUEST was started by admin@internal (Host: HOST.example.com).

2016-06-22 09:42:57,404 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [c63d47b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM GUEST is down with error. Exit message: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'68260611-aa4a-4e79-b093-abcdefghijkl', 'apparentsize': '5368709120', u'imageID': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', u'specParams': {}, u'readonly': u'false', u'domainID': u'97724cca-4600-4508-b234-abcdefghijkl', 'reqsize': '0', u'deviceId': u'06f53064-c1a2-4414-a7d9-abcdefghijkl', 'truesize': '5368709120', u'poolID': u'da7792ba-7398-4577-9a50-bb8b6cfd0fe7', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'}.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments