RHEV: Live Storage Migration fails with "Error code imageErr".

Issue

• When trying to migrate all the disks in a data center to a new storage array to facilitate removal of the old array, some of the VMs in this cluster refuse to snapshot and will not allow their disks to be migrated.

• A Live Storage Migration of disks from one storage domain to another failed.

• The Event Log in the RHEV Admin Portal shows;

2013-Nov-06, 12:22
User admin@internal have failed to move disk guest-x_Disk1 to domain dom-x.

2013-Nov-06, 12:22
User admin@internal moving disk guest-x_Disk1 to domain dom-x.

2013-Nov-06, 12:22
Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'guest-x' has been completed.

2013-Nov-06, 12:22
Failed to create live snapshot 'Auto-generated for Live Storage Migration' for VM 'guest-x'. VM restart is recommended.

2013-Nov-06, 12:22
Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'guest-x' was initiated by admin@internal.

• The /var/log/ovirt-engine/engine.log contains;

2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Failed in VmReplicateDiskStartVDS method
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Error code imageErr and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-16) [4ddef9f5] Command VmReplicateDiskStartVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskStartVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)    
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Failed in VmReplicateDiskFinishVDS method
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-16) [4ddef9f5] Error code imageErr and error message VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file %s could not be found
2013-11-04 19:55:05,431 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-16) [4ddef9f5] Command VmReplicateDiskFinishVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file %s could not be found
RROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)
538_6943:2013-11-04 19:55:05,425 ERROR [org.ovirt.engine.core.bll.lsm.VmReplicateDiskStartTaskHandler] (pool-4-thread-16) [4ddef9f5] Failed VmReplicateDiskStart (Disk 290a70d7-5519-481c-82d2-ed9f08ea0d55 , VM 52e75dc6-20bd-431b-aed4-aff51e336024)

• The /var/log/vdsm/vdsm.log file on the host on which the VM is running contains;

Thread-34::ERROR::2013-11-04 19:52:50,102::blockVolume::405::Storage.Volume::(validateImagePath) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/blockVolume.py", line 403, in validateImagePath
    os.mkdir(imageDir, 0755)
OSError: [Errno 2] No such file or directory: '/rhev/data-center/1278f140-f5f1-4ed0-a97e-d38ef98c0bf3/89bd7b69-3ac2-4895-92bb-b5bd4c551726/images/290a70d7-5519-481c-82d2-ed9f08ea0d55'

Thread-34::ERROR::2013-11-04 19:52:50,102::task::850::TaskManager.Task::(_setError) Task=`fefdbae1-ae95-4d4a-b0ef-149b8a65b121`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 857, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 41, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 3130, in prepareImage
    imgVolumes = img.prepare(sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/image.py", line 364, in prepare
    chain = self.getChain(sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/image.py", line 300, in getChain
    srcVol = volclass(self.repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/blockVolume.py", line 80, in __init__
    volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/volume.py", line 128, in __init__
    self.validate()
  File "/usr/share/vdsm/storage/blockVolume.py", line 89, in validate
    volume.Volume.validate(self)
  File "/usr/share/vdsm/storage/volume.py", line 140, in validate
    self.validateImagePath()
  File "/usr/share/vdsm/storage/blockVolume.py", line 406, in validateImagePath
    raise se.ImagePathError(imageDir)
ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/1278f140-f5f1-4ed0-a97e-d38ef98c0bf3/89bd7b69-3ac2-4895-92bb-b5bd4c551726/images/290a70d7-5519-481c-82d2-ed9f08ea0d55',)

• The /rhev/data-center tree does not contain any Data Center entries;

# tree /rhev/data-center
/rhev/data-center
├── hsm-tasks
└── mnt
    └── blockSD
        .... etc., etc. ....