RHEV: Live Storage Migration fails with "Error code imageErr".
Issue
-
When trying to migrate all the disks in a data center to a new storage array to facilitate removal of the old array, some of the VMs in this cluster refuse to snapshot and will not allow their disks to be migrated.
-
A Live Storage Migration of disks from one storage domain to another failed.
-
The Event Log in the RHEV Admin Portal shows;
2013-Nov-06, 12:22
User admin@internal have failed to move disk guest-x_Disk1 to domain dom-x.
2013-Nov-06, 12:22
User admin@internal moving disk guest-x_Disk1 to domain dom-x.
2013-Nov-06, 12:22
Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'guest-x' has been completed.
2013-Nov-06, 12:22
Failed to create live snapshot 'Auto-generated for Live Storage Migration' for VM 'guest-x'. VM restart is recommended.
2013-Nov-06, 12:22
Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'guest-x' was initiated by admin@internal.
- The
/var/log/ovirt-engine/engine.log
contains;
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Failed in VmReplicateDiskStartVDS method
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Error code imageErr and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-16) [4ddef9f5] Command VmReplicateDiskStartVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Failed in VmReplicateDiskFinishVDS method
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Error code imageErr and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-16) [4ddef9f5] Command VmReplicateDiskFinishVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file %s could not be found
RROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)
538_6943:2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)
- The
/var/log/vdsm/vdsm.log
file on the host on which the VM is running contains;
Thread-34::ERROR::2013-11-04 19:52:50,102::blockVolume::405::Storage.Volume::(validateImagePath) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/blockVolume.py", line 403, in validateImagePath
os.mkdir(imageDir, 0755)
OSError: [Errno 2] No such file or directory: '/rhev/data-center/1278f140-f5f1-4ed0-a97e-d38ef98c0bf3/89bd7b69-3ac2-4895-92bb-b5bd4c551726/images/290a70d7-5519-481c-82d2-ed9f08ea0d55'
Thread-34::ERROR::2013-11-04 19:52:50,102::task::850::TaskManager.Task::(_setError) Task=`fefdbae1-ae95-4d4a-b0ef-149b8a65b121`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 857, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 41, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3130, in prepareImage
imgVolumes = img.prepare(sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/image.py", line 364, in prepare
chain = self.getChain(sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/image.py", line 300, in getChain
srcVol = volclass(self.repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/blockVolume.py", line 80, in __init__
volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/volume.py", line 128, in __init__
self.validate()
File "/usr/share/vdsm/storage/blockVolume.py", line 89, in validate
volume.Volume.validate(self)
File "/usr/share/vdsm/storage/volume.py", line 140, in validate
self.validateImagePath()
File "/usr/share/vdsm/storage/blockVolume.py", line 406, in validateImagePath
raise se.ImagePathError(imageDir)
ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/1278f140-f5f1-4ed0-a97e-d38ef98c0bf3/89bd7b69-3ac2-4895-92bb-b5bd4c551726/images/290a70d7-5519-481c-82d2-ed9f08ea0d55',)
- The
/rhev/data-center
tree does not contain any Data Center entries;
# tree /rhev/data-center
/rhev/data-center
├── hsm-tasks
└── mnt
└── blockSD
.... etc., etc. ....
Environment
- Red Hat Enterprise Virtualization (RHEV) Manager 3.2.3
- Red Hat Enterprise Linux (RHEL) 6.4 hosts;
- vdsm-4.10.2-27.0
- libvirt-0.10.2-18.el6_4.9
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.