Snapshot delete fails due to incorrect image chain.

Solution In Progress - Updated -

Environment

  • Red Hat Virtualization (RHV) 4.4 SP1
    • ovirt-engine-4.5.3.7-1.el8ev.noarch
    • vdsm-4.50.3.7-1.el8ev.x86_64
  • vProtect backup software

Issue

  • Snapshot delete fails leaving incorrect disk image chain.

  • In vdsm.log we can see that it fails because a volume is not present in the backing file chain:

libvirt.libvirtError: internal error: unable to execute QEMU command 'block-commit': 'libvirt-644-format' is not in this backing file chain

Resolution

  • Please open a case with Red Hat Support as correcting the VM will require reviewing data/logs.

Root Cause

  • The issue has been reported in BZ 2254096 and it's currently being investigated.

Diagnostic Steps

  • According to RHV-M DB, the following three images/volumes are associated with VM disk.
              image_guid              |            image_group_id            |            vm_snapshot_id            |               parentid               | imagestatus | volume_type | volume_format | active 
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------+-------------+---------------+--------
 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 | ead3fca9-58b9-41fa-873f-43dfb0b59042 | 14fedab1-210b-4d1c-9ce9-748875a537ba | 00000000-0000-0000-0000-000000000000 |           1 |           2 |             4 | f
 94b6e47c-42c0-429b-85fd-97f2324f1240 | ead3fca9-58b9-41fa-873f-43dfb0b59042 | b8a9e9ba-1ba1-4153-85f5-d2042f59bd81 | 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 |           1 |           2 |             4 | f
 0f4e4940-bafd-4f36-a344-17c24dc28035 | ead3fca9-58b9-41fa-873f-43dfb0b59042 | bb2872d7-a874-44f9-86e8-f2a709fa5b0b | 94b6e47c-42c0-429b-85fd-97f2324f1240 |           1 |           2 |             4 | t
(3 rows)
  • In the engine logs, the correlation id a14b0bc8-b898-4e49-89f2-d22d7de3fe81 is for merging images/volume 94b6e47c-42c0-429b-85fd-97f2324f1240 into 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 which has failed.
2023-11-23 08:44:57,857+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-7) [a14b0bc8-b898-4e49-89f2-d22d7de3fe81] START, MergeVDSCommand(HostName = my-host, MergeVDSCommandParameters:{hostId='41568e2a-3c9f-4e1a-aa31-f166ff0edd0c', vmId='8f4f5d30-d6a0-402d-bac9-6ed430ea11b3', storagePoolId='46a08fb6-d652-4890-8d9d-98d4fa35497c', storageDomainId='1394e07f-5318-443a-a291-5d06af98d475', imageGroupId='ead3fca9-58b9-41fa-873f-43dfb0b59042', imageId='0f4e4940-bafd-4f36-a344-17c24dc28035', baseImageId='4aceea12-8c1a-43e0-a2e0-4e8b690ec026', topImageId='94b6e47c-42c0-429b-85fd-97f2324f1240', bandwidth='0'}), log id: 382ee607

2023-11-23 08:46:18,458+01 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-5) [a14b0bc8-b898-4e49-89f2-d22d7de3fe81] Failed to live merge. Top volume 94b6e47c-42c0-429b-85fd-97f2324f1240 is still in qemu chain [94b6e47c-42c0-429b-85fd-97f2324f1240, 0f4e4940-bafd-4f36-a344-17c24dc28035, 4aceea12-8c1a-43e0-a2e0-4e8b690ec026]

2023-11-23 08:46:54,640+01 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [a14b0bc8-b898-4e49-89f2-d22d7de3fe81] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2023-11-23 08:46:54,708+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-61) [a14b0bc8-b898-4e49-89f2-d22d7de3fe81] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2023-11-22 07:27:53.518008' for VM 'MY_VM'.
  • Checking on the host where the VM is running and snapshot delete (merge) was initiated, we see the "START merge" for images/volume for baseVolUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026' and topVolUUID='94b6e47c-42c0-429b-85fd-97f2324f1240'
2023-11-23 08:44:57,858+0100 INFO  (jsonrpc/4) [api.virt] START merge(drive={'poolID': '46a08fb6-d652-4890-8d9d-98d4fa35497c', 'volumeID': '0f4e4940-bafd-4f36-a344-17c24dc28035', 'imageID': 'ead3fca9-58b9-41fa-873f-43dfb0b59042', 'domainID': '1394e07f-5318-443a-a291-5d06af98d475'}, baseVolUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026', topVolUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', bandwidth='0', jobUUID='cc72a8d5-4617-47c1-98d6-902f61c7459f') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, vmId=8f4f5d30-d6a0-402d-bac9-6ed430ea11b3 (api:31)

2023-11-23 08:44:57,859+0100 INFO  (jsonrpc/4) [vdsm.api] START getVolumeInfo(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', spUUID='46a08fb6-d652-4890-8d9d-98d4fa35497c', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=1e7a1918-f565-40a8-a165-33804d8dff66 (api:31)
2023-11-23 08:44:58,024+0100 INFO  (jsonrpc/4) [vdsm.api] FINISH getVolumeInfo return={'info': {'uuid': '4aceea12-8c1a-43e0-a2e0-4e8b690ec026', 'type': 'SPARSE', 'format': 'COW', 'disktype': 'DATA', 'voltype': 'INTERNAL', 'capacity': '37580963840', 'parent': '00000000-0000-0000-0000-000000000000', 'description': '', 'pool': '', 'domain': '1394e07f-5318-443a-a291-5d06af98d475', 'image': 'ead3fca9-58b9-41fa-873f-43dfb0b59042', 'ctime': '1647874226', 'mtime': '0', 'legality': 'LEGAL', 'generation': 0, 'sequence': 0, 'apparentsize': '37580963840', 'truesize': '37580963840', 'status': 'OK', 'lease': {'path': '/dev/1394e07f-5318-443a-a291-5d06af98d475/leases', 'offset': 178257920, 'owners': [], 'version': None}, 'children': []}} from=::ffff:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=1e7a1918-f565-40a8-a165-33804d8dff66 (api:37)

2023-11-23 08:44:58,025+0100 INFO  (jsonrpc/4) [vdsm.api] START getVolumeInfo(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', spUUID='46a08fb6-d652-4890-8d9d-98d4fa35497c', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='94b6e47c-42c0-429b-85fd-97f2324f1240') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=70bc8c5c-984b-4e11-91b0-dd388546ec9f (api:31)
2023-11-23 08:44:58,049+0100 INFO  (jsonrpc/4) [vdsm.api] FINISH getVolumeInfo return={'info': {'uuid': '94b6e47c-42c0-429b-85fd-97f2324f1240', 'type': 'SPARSE', 'format': 'COW', 'disktype': 'DATA', 'voltype': 'INTERNAL', 'capacity': '37580963840', 'parent': '4aceea12-8c1a-43e0-a2e0-4e8b690ec026', 'description': '', 'pool': '', 'domain': '1394e07f-5318-443a-a291-5d06af98d475', 'image': 'ead3fca9-58b9-41fa-873f-43dfb0b59042', 'ctime': '1700634477', 'mtime': '0', 'legality': 'LEGAL', 'generation': 0, 'sequence': 7, 'apparentsize': '8589934592', 'truesize': '8589934592', 'status': 'OK', 'lease': {'path': '/dev/1394e07f-5318-443a-a291-5d06af98d475/leases', 'offset': 312475648, 'owners': [], 'version': None}, 'children': []}} from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=70bc8c5c-984b-4e11-91b0-dd388546ec9f (api:37)

2023-11-23 08:44:58,050+0100 INFO  (jsonrpc/4) [vdsm.api] START prune_bitmaps(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', baseUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=d9cb67e3-45ac-41d9-a47c-291e4ab81f15 (api:31)
2023-11-23 08:44:58,319+0100 INFO  (jsonrpc/4) [vdsm.api] FINISH prune_bitmaps return=None from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=d9cb67e3-45ac-41d9-a47c-291e4ab81f15 (api:37)

2023-11-23 08:44:58,319+0100 INFO  (jsonrpc/4) [vdsm.api] START measure(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', dest_format=4, backing=True, baseUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=cb90c2cb-a2b3-4cba-910b-fec584c236fb (api:31)
2023-11-23 08:44:58,442+0100 INFO  (jsonrpc/4) [vdsm.api] FINISH measure return={'result': {'bitmaps': 1245184, 'required': 37578539008, 'fully-allocated': 37586927616}} from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=a14b0bc8-b898-4e49-89f2-d22d7de3fe81, task_id=cb90c2cb-a2b3-4cba-910b-fec584c236fb (api:37)

2023-11-23 08:44:58,443+0100 INFO  (jsonrpc/4) [virt.livemerge] Job cc72a8d5-4617-47c1-98d6-902f61c7459f switching state from INIT to COMMIT (livemerge:155)
  • The original chain reported 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 < 94b6e47c-42c0-429b-85fd-97f2324f1240 < 0f4e4940-bafd-4f36-a344-17c24dc28035.
2023-11-23 08:44:58,466+0100 INFO  (jsonrpc/4) [virt.livemerge] Starting merge with job_id='cc72a8d5-4617-47c1-98d6-902f61c7459f', original chain=4aceea12-8c1a-43e0-a2e0-4e8b690ec026 < 94b6e47c-42c0-429b-85fd-97f2324f1240 < 0f4e4940-bafd-4f36-a344-17c24dc28035 (top), disk='vdb', base='vdb[18]', top='vdb[644]', bandwidth=0, flags=8 (livemerge:401)
2023-11-23 08:45:38,453+0100 INFO  (libvirt/events) [virt.vm] (vmId='8f4f5d30-d6a0-402d-bac9-6ed430ea11b3') Block job cc72a8d5-4617-47c1-98d6-902f61c7459f type COMMIT for drive vdb has completed (vm:5952)
  • No space error reported in the /var/log/messages.
Nov 23 08:45:38 my-host journal[9132]: internal error: unable to execute QEMU command 'blockdev-reopen': Failed to update bitmap extension: No space left on device
  • Job completes and the newChain is the same as the original chain.
    newChain=['4aceea12-8c1a-43e0-a2e0-4e8b690ec026', '94b6e47c-42c0-429b-85fd-97f2324f1240', '0f4e4940-bafd-4f36-a344-17c24dc28035']
2023-11-23 08:45:46,154+0100 INFO  (periodic/4) [virt.livemerge] Job cc72a8d5-4617-47c1-98d6-902f61c7459f has completed (livemerge:735)
2023-11-23 08:45:46,154+0100 INFO  (periodic/4) [virt.livemerge] Job cc72a8d5-4617-47c1-98d6-902f61c7459f switching state from COMMIT to CLEANUP (livemerge:155)
2023-11-23 08:45:46,163+0100 INFO  (periodic/4) [virt.livemerge] Starting cleanup for job cc72a8d5-4617-47c1-98d6-902f61c7459f (livemerge:788)
2023-11-23 08:45:46,468+0100 INFO  (merge/cc72a8d5) [virt.vm] (vmId='8f4f5d30-d6a0-402d-bac9-6ed430ea11b3') Synchronizing volume chain after live merge (job cc72a8d5-4617-47c1-98d6-902f61c7459f) (livemerge:975)
2023-11-23 08:45:46,477+0100 INFO  (merge/cc72a8d5) [vdsm.api] START imageSyncVolumeChain(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='0f4e4940-bafd-4f36-a344-17c24dc28035', newChain=['4aceea12-8c1a-43e0-a2e0-4e8b690ec026', '94b6e47c-42c0-429b-85fd-97f2324f1240', '0f4e4940-bafd-4f36-a344-17c24dc28035']) from=internal, task_id=01a16ece-39fe-42ef-94ec-a4b07ecceadd (api:31)
2023-11-23 08:45:46,757+0100 INFO  (merge/cc72a8d5) [storage.image] Current chain=4aceea12-8c1a-43e0-a2e0-4e8b690ec026 < 94b6e47c-42c0-429b-85fd-97f2324f1240 < 0f4e4940-bafd-4f36-a344-17c24dc28035 (top)  (image:882)
2023-11-23 08:45:46,767+0100 INFO  (merge/cc72a8d5) [virt.vm] (vmId='8f4f5d30-d6a0-402d-bac9-6ed430ea11b3') Synchronization completed (job cc72a8d5-4617-47c1-98d6-902f61c7459f) (livemerge:984)
  • Subsequent snapshot operation fails.
2023-11-23 18:48:37,742+0100 INFO  (jsonrpc/5) [api.virt] START merge(drive={'poolID': '46a08fb6-d652-4890-8d9d-98d4fa35497c', 'volumeID': '0f4e4940-bafd-4f36-a344-17c24dc28035', 'imageID': 'ead3fca9-58b9-41fa-873f-43dfb0b59042', 'domainID': '1394e07f-5318-443a-a291-5d06af98d475'}, baseVolUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026', topVolUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', bandwidth='0', jobUUID='41fc9054-e89a-43a4-8d40-8aeccc5b3709') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=0b796641-2357-4389-b148-e31d72750220, vmId=8f4f5d30-d6a0-402d-bac9-6ed430ea11b3 (api:31)
2023-11-23 18:48:37,925+0100 INFO  (jsonrpc/5) [vdsm.api] START prune_bitmaps(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', baseUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=0b796641-2357-4389-b148-e31d72750220, task_id=7a3a69c9-956a-4eff-b9f2-d0c00ac48539 (api:31)
2023-11-23 18:48:38,171+0100 INFO  (jsonrpc/5) [vdsm.api] FINISH prune_bitmaps return=None from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=0b796641-2357-4389-b148-e31d72750220, task_id=7a3a69c9-956a-4eff-b9f2-d0c00ac48539 (api:37)
2023-11-23 18:48:38,172+0100 INFO  (jsonrpc/5) [vdsm.api] START measure(sdUUID='1394e07f-5318-443a-a291-5d06af98d475', imgUUID='ead3fca9-58b9-41fa-873f-43dfb0b59042', volUUID='94b6e47c-42c0-429b-85fd-97f2324f1240', dest_format=4, backing=True, baseUUID='4aceea12-8c1a-43e0-a2e0-4e8b690ec026') from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=0b796641-2357-4389-b148-e31d72750220, task_id=3864b88a-1e94-4a61-9301-cfff410e12de (api:31)
2023-11-23 18:48:38,241+0100 INFO  (jsonrpc/5) [vdsm.api] FINISH measure return={'result': {'bitmaps': 1245184, 'required': 37578539008, 'fully-allocated': 37586927616}} from=::xxxx:XX.XXX.XXX.XX,12345, flow_id=0b796641-2357-4389-b148-e31d72750220, task_id=3864b88a-1e94-4a61-9301-cfff410e12de (api:37)
2023-11-23 18:48:38,242+0100 INFO  (jsonrpc/5) [virt.livemerge] Job 41fc9054-e89a-43a4-8d40-8aeccc5b3709 switching state from INIT to COMMIT (livemerge:155)
2023-11-23 18:48:38,260+0100 INFO  (jsonrpc/5) [virt.livemerge] Starting merge with job_id='41fc9054-e89a-43a4-8d40-8aeccc5b3709', original chain=4aceea12-8c1a-43e0-a2e0-4e8b690ec026 < 94b6e47c-42c0-429b-85fd-97f2324f1240 < 0f4e4940-bafd-4f36-a344-17c24dc28035 (top), disk='vdb', base='vdb[18]', top='vdb[644]', bandwidth=0, flags=8 (livemerge:401)
2023-11-23 18:48:38,295+0100 ERROR (jsonrpc/5) [api] FINISH merge error=Merge failed: {'job': '41fc9054-e89a-43a4-8d40-8aeccc5b3709', 'reason': "internal error: unable to execute QEMU command 'block-commit': 'libvirt-644-format' is not in this backing file chain"} (api:114)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 410, in _start_commit
    flags=flags)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 162, in call
    return getattr(self._vm._dom, name)(*a, **kw)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 104, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 114, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 78, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 919, in blockCommit
    raise libvirtError('virDomainBlockCommit() failed')
libvirt.libvirtError: internal error: unable to execute QEMU command 'block-commit': 'libvirt-644-format' is not in this backing file chain

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 107, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 764, in merge
    drive, baseVolUUID, topVolUUID, bandwidth, jobUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5969, in merge
    driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 307, in merge
    self._start_commit(drive, job)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 413, in _start_commit
    raise exception.MergeFailed(str(e), job=job.id)
vdsm.common.exception.MergeFailed: Merge failed: {'job': '41fc9054-e89a-43a4-8d40-8aeccc5b3709', 'reason': "internal error: unable to execute QEMU command 'block-commit': 'libvirt-644-format' is not in this backing file chain"}
  • VM XML info also shows three images/volume in chain.
</disk>
<disk type='block' device='disk' snapshot='no'>
  <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native'/>

  <source dev='/rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/0f4e4940-bafd-4f36-a344-17c24dc28035' index='651'>
    <seclabel model='dac' relabel='no'/>
  </source>
  <backingStore type='block' index='644'>
    <format type='qcow2'/>

    <source dev='/rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/94b6e47c-42c0-429b-85fd-97f2324f1240'>
      <seclabel model='dac' relabel='no'/>
    </source>
    <backingStore type='block' index='18'>
      <format type='qcow2'/>

      <source dev='/rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/4aceea12-8c1a-43e0-a2e0-4e8b690ec026'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore/>
    </backingStore>
  </backingStore>
  <target dev='vdb' bus='virtio'/>
  <serial>ead3fca9-58b9-41fa-873f-43dfb0b59042</serial>
  <alias name='ua-ead3fca9-58b9-41fa-873f-43dfb0b59042'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk
  • All the lv's are open and active on the Storage domain.
  4aceea12-8c1a-43e0-a2e0-4e8b690ec026 1394e07f-5318-443a-a291-5d06af98d475    1 -wi-ao----  35.00g   253   79  IU_ead3fca9-58b9-41fa-873f-43dfb0b59042,MD_70,PU_00000000-0000-0000-0000-000000000000 
  94b6e47c-42c0-429b-85fd-97f2324f1240 1394e07f-5318-443a-a291-5d06af98d475    3 -wi-ao----   8.00g   253   99  IU_ead3fca9-58b9-41fa-873f-43dfb0b59042,MD_198,PU_4aceea12-8c1a-43e0-a2e0-4e8b690ec026
  0f4e4940-bafd-4f36-a344-17c24dc28035 1394e07f-5318-443a-a291-5d06af98d475    1 -wi-ao----   2.00g   253  111  IU_ead3fca9-58b9-41fa-873f-43dfb0b59042,MD_92,PU_94b6e47c-42c0-429b-85fd-97f2324f1240
  • The dm devices are also associated to the VM pid.
lrwxrwxrwx.  1 0 0       8 Nov 28 09:00 1394e07f--5318--443a--a291--5d06af98d475-4aceea12--8c1a--43e0--a2e0--4e8b690ec026 -> ../dm-79
lrwxrwxrwx.  1 0 0       8 Nov 28 18:23 1394e07f--5318--443a--a291--5d06af98d475-94b6e47c--42c0--429b--85fd--97f2324f1240 -> ../dm-99
lrwxrwxrwx.  1 0 0       9 Nov 28 09:00 1394e07f--5318--443a--a291--5d06af98d475-0f4e4940--bafd--4f36--a344--17c24dc28035 -> ../dm-111
  • VM pid is 1892562 and all the dm devices are attached to the VM pid.
$ cat ps_auxwwwm |grep qemu |grep 1892562
qemu     1892562 29.3  5.4 18461516 14256248 ?   -    Aug08 47889:02 /usr/libexec/qemu-kvm -name guest=MY_VM

$ cat lsof_M_-n_-l_-c  |grep dm-79
qemu-kvm  1892562      107   19u      BLK             253,79       0t0 2088813453 /dev/dm-79

$ cat lsof_M_-n_-l_-c  |grep dm-99
qemu-kvm  1892562      107   14r      BLK             253,99       0t0   37175290 /dev/dm-99

$ cat lsof_M_-n_-l_-c  |grep dm-111
qemu-kvm  1892562      107  136u      BLK            253,111       0t0   59397694 /dev/dm-111
  • The backing chain shows backing file 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 for the top image/volume 0f4e4940-bafd-4f36-a344-17c24dc28035.
$ cat virsh_-r_domblklist_MY_VM |grep ead3fca9-58b9-41fa-873f-43dfb0b59042
 vdb      /rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/0f4e4940-bafd-4f36-a344-17c24dc28035

-----

# qemu-img info --backing-chain /dev/1394e07f-5318-443a-a291-5d06af98d475/0f4e4940-bafd-4f36-a344-17c24dc28035
image: /dev/1394e07f-5318-443a-a291-5d06af98d475/0f4e4940-bafd-4f36-a344-17c24dc28035
file format: qcow2
virtual size: 35 GiB (37580963840 bytes)
disk size: 0 B
cluster_size: 65536
backing file: 4aceea12-8c1a-43e0-a2e0-4e8b690ec026 (actual path: /dev/1394e07f-5318-443a-a291-5d06af98d475/4aceea12-8c1a-43e0-a2e0-4e8b690ec026)
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

image: /dev/1394e07f-5318-443a-a291-5d06af98d475/4aceea12-8c1a-43e0-a2e0-4e8b690ec026
file format: qcow2
virtual size: 35 GiB (37580963840 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: in-use
                [1]: auto
            name: 75c31316-870a-4223-9561-78ae89d9f08d
            granularity: 65536
        [1]:
            flags:
                [0]: in-use
                [1]: auto
            name: 3d3eb83a-401a-4517-8bb0-7b032e94c5ac
            granularity: 65536
        [2]:
            flags:
                [0]: in-use
                [1]: auto
            name: 22bb335f-3624-4520-a645-927f141e3627
            granularity: 65536
        [3]:
            flags:
                [0]: in-use
                [1]: auto
            name: f1efa52a-8c04-4a21-b97f-214ec92295c9
            granularity: 65536
        [4]:
            flags:
                [0]: in-use
                [1]: auto
            name: eb4e9195-81a4-4347-83d4-2967a37e665c
            granularity: 65536
        [5]:
            flags:
                [0]: in-use
                [1]: auto
            name: d60e50f8-dc48-4861-9529-4c33c1375639
            granularity: 65536
    refcount bits: 16
    corrupt: false
    extended l2: false
  • virsh qemu-monitor-command MY_VM --hmp --cmd 'info block'. Only shows depth chain 1.
    94b6e47c-42c0-429b-85fd-97f2324f1240 is not seen in the chain.
libvirt-651-format: json:{"backing": {"driver": "qcow2", "file": {"driver": "host_device", "filename": "/rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/4aceea12-8c1a-43e0-a2e0-4e8b690ec026"}}, "driver": "qcow2", "file": {"driver": "host_device", "filename": "/rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/0f4e4940-bafd-4f36-a344-17c24dc28035"}} (qcow2)
    Attached to:      /machine/peripheral/ua-ead3fca9-58b9-41fa-873f-43dfb0b59042/virtio-backend
    Cache mode:       writeback, direct
    Backing file:     /rhev/data-center/mnt/blockSD/1394e07f-5318-443a-a291-5d06af98d475/images/ead3fca9-58b9-41fa-873f-43dfb0b59042/4aceea12-8c1a-43e0-a2e0-4e8b690ec026 (chain depth: 1)
  • The dm-99 device shows no activity when dd'ing the drive. Only the base and top dm devices show activity.
08:48:36 AM dev253-79    256.00 169729.50      0.00    663.01      0.42      1.66      1.90     48.70
08:48:36 AM dev253-99      1.00      0.50      0.00      0.50      0.00      0.00      1.00      0.10
08:48:36 AM dev253-111    570.00  69056.00      0.00    121.15      0.57      1.00      0.48     27.50

08:48:37 AM dev253-79    561.00 221120.00      0.00    394.15      0.69      1.23      1.36     76.50
08:48:37 AM dev253-99      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:48:37 AM dev253-111   2064.00 194624.00      0.00     94.29      1.91      0.92      0.35     71.80

08:48:38 AM dev253-79    482.00 408320.00      0.00    847.14      0.80      1.67      1.88     90.70
08:48:38 AM dev253-99      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:48:38 AM dev253-111    261.00  66880.00      0.00    256.25      0.31      1.18      0.95     24.90

08:48:39 AM dev253-79    364.00 353024.00      0.00    969.85      0.72      1.98      2.27     82.80
08:48:39 AM dev253-99      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:48:39 AM dev253-111    203.00  86272.00     27.50    425.12      0.28      1.38      1.09     22.20

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments