Guest import failed from NFS export domain

Solution Verified - Updated -

Environment

Red Hat Enterprise Virtualization (RHEV) 3.3

Issue

While trying to import a previously exported guest, the import failed and the following error message was reported on the Webadmin portal

 Failed to import Vm VM-TEST  to Data Center DC-TEST, Cluster CLUSTER-TEST

The message ImageDoesNotExistInSD?: Image does not exist in domain is logged in the engine.log file.

Resolution

  1. Detach NFS export domain
  2. Attach NFS export domain
  3. Import the guest from the export domain

Root Cause

  • The image does not exist on domain error message points to missing storage domain links under /rhev/data-center/SPUUID directory.
  • Storage Domain links couldn't be verified since the sosreport provided was after the problem was resolved.
  • The environment had experienced network issues which had caused the Data Center to change to non-responsive state.

Diagnostic Steps

vdsm.log:
Metadata collection error message seem to indicate a failed attempt to access the storage domain.
As a result the import of the guest failed due to non-existent image on the storage domain

Thread-3092351::WARNING::2014-11-04 10:21:12,189::fileSD::622::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/xxxxxxxx.xxxx.xx-xxx.local:_rhev-export-domain timedout
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/fileSD.py", line 611, in collectMetaFiles
    sd.DOMAIN_META_DATA))
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 297, in callCrabRPCFunction
    *args, **kwargs)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 184, in callCrabRPCFunction
    rawLength = self._recvAll(LENGTH_STRUCT_LENGTH, timeout)
  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 150, in _recvAll
    raise Timeout()
Timeout
.......

Thread-3092565::DEBUG::2014-11-04 10:28:46,921::task::579::TaskManager.Task::(_updateState) Task=`79732675-f3a6-477b-955f-1682f3abd147`::moving from state init -> state preparing
Thread-3092565::INFO::2014-11-04 10:28:46,921::logUtils::44::dispatcher::(wrapper) Run and protect: copyImage(sdUUID='977b3c85-10a3-42e1-886c-96aab25d828c', spUUID='2f2b9f2d-a761-4043-bf06-898c06d0eae7', vmUUID='', srcImgUUID='93f3d938-6042-42ea-a868-07f96be15dc3', srcVolUUID='c857098f-47d5-4bf1-a241-7a5d8363f103', dstImgUUID='93f3d938-6042-42ea-a868-07f96be15dc3', dstVolUUID='c857098f-47d5-4bf1-a241-7a5d8363f103', description='', dstSdUUID='0d8d1740-7aac-4a44-bff3-e8da9781eda6', volType=8, volFormat=5, preallocate=1, postZero='false', force='true')
Thread-3092565::INFO::2014-11-04 10:28:46,922::image::285::Storage.Image::(isLegal) image 93f3d938-6042-42ea-a868-07f96be15dc3 in domain 977b3c85-10a3-42e1-886c-96aab25d828c has vollist []
Thread-3092565::ERROR::2014-11-04 10:28:46,923::task::850::TaskManager.Task::(_setError) Task=`79732675-f3a6-477b-955f-1682f3abd147`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 857, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1778, in copyImaIssues with fiber storage on RHEV platformge
    pool.validateVolumeChain(sdUUID, srcImgUUID)
  File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 2023, in validateVolumeChain
    image.Image(self.poolPath).validateVolumeChain(sdUUID, imgUUID)
  File "/usr/share/vdsm/storage/image.py", line 625, in validateVolumeChain
    chain = self.getChain(sdUUID, imgUUID)
  File "/usr/share/vdsm/storage/image.py", line 185, in getChain
    raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
ImageDoesNotExistInSD?: Image does not exist in domain: 'image=93f3d938-6042-42ea-a868-07f96be15dc3, domain=/b3c85-10a3-42e1-886c-96aab25d828c'
Thread-3092565::DEBUG::2014-11-04 10:28:46,926::task::869::TaskManager.Task::(_run) Task=`79732675-f3a6-477b-955f-1682f3abd147`::Task._run: 79732675-f3a6-477b-955f-1682f3abd147 ('977b3c85-10a3-42e1-886c-96aab25d828c', '2f2b9f2d-a761-4043-bf06-898c06d0eae7', '', '93f3d938-6042-42ea-a868-07f96be15dc3', 'c857098f-47d5-4bf1-a241-7a5d8363f103', '93f3d938-6042-42ea-a868-07f96be15dc3', 'c857098f-47d5-4bf1-a241-7a5d8363f103', '', '0d8d1740-7aac-4a44-bff3-e8da9781eda6', 8, 5, 1, 'false', 'true') {} failed - stopping task
Thread-3092565::DEBUG::2014-11-04 10:28:46,926::task::1194::TaskManager.Task::(stop) Task=`79732675-f3a6-477b-955f-1682f3abd147`::stopping in state preparing (force False)
Thread-3092565::DEBUG::2014-11-04 10:28:46,926::task::974::TaskManager.Task::(_decref) Task=`79732675-f3a6-477b-955f-1682f3abd147`::ref 1 aborting True
Thread-3092565::INFO::2014-11-04 10:28:46,927::task::1151::TaskManager.Task::(prepare) Task=`79732675-f3a6-477b-955f-1682f3abd147`::aborting: Task is aborted: 'Image does not exist in domain' - code 268
Thread-3092565::DEBUG::2014-11-04 10:28:46,927::task::1156::TaskManager.Task::(prepare) Task=`79732675-f3a6-477b-955f-1682f3abd147`::Prepare: aborted: Image does not exist in domain
Thread-3092565::DEBUG::2014-11-04 10:28:46,927::task::974::TaskManager.Task::(_decref) Task=`79732675-f3a6-477b-955f-1682f3abd147`::ref 0 aborting True
Thread-3092565::DEBUG::2014-11-04 10:28:46,927::task::909::TaskManager.Task::(_doAbort) Task=`79732675-f3a6-477b-955f-1682f3abd147`::Task._doAbort: force False
Thread-3092565::DEBUG::2014-11-04 10:28:46,927::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-3092565::DEBUG::2014-11-04 10:28:46,927::task::579::TaskManager.Task::(_updateState) Task=`79732675-f3a6-477b-955f-1682f3abd147`::moving from state preparing -> state aborting
Thread-3092565::DEBUG::2014-11-04 10:28:46,928::task::534::TaskManager.Task::(__state_aborting) Task=`79732675-f3a6-477b-955f-1682f3abd147`::_aborting: recover policy none
Thread-3092565::DEBUG::2014-11-04 10:28:46,928::task::579::TaskManager.Task::(_updateState) Task=`79732675-f3a6-477b-955f-1682f3abd147`::moving from state aborting -> state failed

engine.log:
The refresh of VDS failed due to network errors.
As a result when the Import of the guest was initiated the CopyImageVDS execution failed.

2014-11-04 10:18:59,824 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Failed to refresh VDS , vds = 48e3ed1e-d2c7-40e7-83f7-4af194d02fe7 :       X-XXXX-1, VDS Network Error, continuing.
java.net.SocketTimeoutException: connect timed out
2014-11-04 10:19:00,032 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) Failed to refresh VDS , vds = 4e5f52c0-392f-4c73-96cf-364fb33eed4c :       X-XXXX 2, VDS Network Error, continuing.
java.net.SocketTimeoutException: connect timed out
......

2014-11-04 10:25:09,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-25) FINISH, ConnectStoragePoolVDSCommand, log id: 139c8e3f
2014-11-04 10:25:09,921 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-25) Domain 977b3c85-10a3-42e1-886c-96aab25d828c:ExportDomain was reported       with error code 358
2014-11-04 10:25:09,923 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-25) Storage Domain ExportDomain of pool TEST-DC is in problem in host i-mpapp2

.......

2014-11-04 10:28:29,771 INFO  [org.ovirt.engine.core.bll.ImportVmCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] Lock Acquired to object EngineLock [exclusiveLocks= key: i-app131 value: VM_NAME
, sharedLocks= key: 2f00944d-c36d-4fb3-8a09-d720a48bb775 value: REMOTE_VM
]
2014-11-04 10:28:29,815 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] START, GetVmsInfoVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, storageDomainId = 977b3c85-10a3-42e1-886c-96aab25d828c, vmIdList = null), log id: 2b7f2d1
2014-11-04 10:28:30,339 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] FINISH, GetVmsInfoVDSCommand, log id: 2b7f2d1
2014-11-04 10:28:30,615 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] START, GetImageDomainsListVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, imageGroupId = 93f3d938-6042-42ea-a868-07f96be15dc3), log id: 4f6b701a
2014-11-04 10:28:35,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] FINISH, GetImageDomainsListVDSCommand, return: [], log id: 4f6b701a
2014-11-04 10:28:35,386 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] START, GetImageDomainsListVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, imageGroupId = 0c40bb73-850c-4a3d-9e29-6cc71f9ca6d5), log id: 7b1d4ef5
2014-11-04 10:28:42,464 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] FINISH, GetImageDomainsListVDSCommand, return: [], log id: 7b1d4ef5
2014-11-04 10:28:42,465 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] START, DoesImageExistVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, storageDomainId = 977b3c85-10a3-42e1-886c-96aab25d828c, imageGroupId = 0c40bb73-850c-4a3d-9e29-6cc71f9ca6d5, imageId = 15e4687d-da73-47e2-b5dd-40e0e39d523f), log id: 59526999
2014-11-04 10:28:43,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] FINISH, DoesImageExistVDSCommand, return: true, log id: 59526999
2014-11-04 10:28:43,764 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] START, DoesImageExistVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, storageDomainId = 977b3c85-10a3-42e1-886c-96aab25d828c, imageGroupId = 93f3d938-6042-42ea-a868-07f96be15dc3, imageId = c857098f-47d5-4bf1-a241-7a5d8363f103), log id: 7a5169c8
2014-11-04 10:28:44,326 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-4) [2ccc6ed0] FINISH, DoesImageExistVDSCommand, return: true, log id: 7a5169c8
2014-11-04 10:28:44,734 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-8) START, GetVmsInfoVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, storageDomainId = 977b3c85-10a3-42e1-886c-96aab25d828c, vmIdList = null), log id: 20963610
2014-11-04 10:28:44,772 INFO  [org.ovirt.engine.core.bll.ImportVmCommand] (pool-4-thread-30) [2ccc6ed0] Running command: ImportVmCommand internal: false. Entities affected :  ID: 0d8d1740-7aac-4a44-bff3-e8da9781eda6 Type: Storage,  ID: 0d8d1740-7aac-4a44-bff3-e8da9781eda6 Type: Storage
2014-11-04 10:28:44,948 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-8) FINISH, GetVmsInfoVDSCommand, log id: 20963610
2014-11-04 10:28:46,488 INFO  [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-30) [341759a6] Running command: CopyImageGroupCommand internal: true. Entities affected :  ID: 0d8d1740-7aac-4a44-bff3-e8da9781eda6 Type: Storage
2014-11-04 10:28:46,699 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-30) [341759a6] START, CopyImageVDSCommand( storagePoolId = 2f2b9f2d-a761-4043-bf06-898c06d0eae7, ignoreFailoverLimit = false, storageDomainId = 977b3c85-10a3-42e1-886c-96aab25d828c, imageGroupId = 93f3d938-6042-42ea-a868-07f96be15dc3, imageId = c857098f-47d5-4bf1-a241-7a5d8363f103, dstImageGroupId = 93f3d938-6042-42ea-a868-07f96be15dc3, vmId = 2f00944d-c36d-4fb3-8a09-d720a48bb775, dstImageId = c857098f-47d5-4bf1-a241-7a5d8363f103, imageDescription = , dstStorageDomainId = 0d8d1740-7aac-4a44-bff3-e8da9781eda6, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Preallocated, postZero = false, force = true), log id: 14956448
2014-11-04 10:28:46,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-30) [341759a6] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID
2014-11-04 10:28:46,700 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-30) [341759a6] -- copyImage parameters:
                sdUUID=977b3c85-10a3-42e1-886c-96aab25d828c
                spUUID=2f2b9f2d-a761-4043-bf06-898c06d0eae7
                vmGUID=2f00944d-c36d-4fb3-8a09-d720a48bb775
                srcImageGUID=93f3d938-6042-42ea-a868-07f96be15dc3
                srcVolUUID=c857098f-47d5-4bf1-a241-7a5d8363f103
                dstImageGUID=93f3d938-6042-42ea-a868-07f96be15dc3
                dstVolUUID=c857098f-47d5-4bf1-a241-7a5d8363f103
                descr=
                dstSdUUID=0d8d1740-7aac-4a44-bff3-e8da9781eda6
2014-11-04 10:28:46,995 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-30) [341759a6] Command CopyImageVDS execution failed. Exception: IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Image does not exist in domain: 'image=93f3d938-6042-42ea-a868-07f96be15dc3, domain=977b3c85-10a3-42e1-886c-96aab25d828c'
2014-11-04 10:28:46,995 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-30) [341759a6] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 14956448
2014-11-04 10:28:46,995 ERROR [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-30) [341759a6] Command org.ovirt.engine.core.bll.CopyImageGroupCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailoverException: IRSGenericException: IRSErrorException: Image does not exist in domain: 'image=93f3d938-6042-42ea-a868-07f96be15dc3, domain=977b3c85-10a3-42e1-886c-96aab25d828c' (Failed with error ImageDoesNotExistInDomainError and code 268)
2014-11-04 10:28:47,140 ERROR [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-30) [341759a6] Transaction rolled-back for command: org.ovirt.engine.core.bll.CopyImageGroupCommand.
2014-11-04 10:28:47,140 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (pool-4-thread-30) [341759a6] transaction rolled back
2014-11-04 10:28:47,140 ERROR [org.ovirt.engine.core.bll.ImportVmCommand] (pool-4-thread-30) [341759a6] Command org.ovirt.engine.core.bll.ImportVmCommand throw exception: javax.ejb.EJBTransactionRolledbackException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.