After a VM's disks were moved from one Storage Domain to another, the VM failed to start and reported "Failed to get size for volume"
Environment
- Red Hat Enterprise Virtualization (RHEV) 3.4
Issue
-
Disk migration ends with Failed to get size for volume.
-
After an offline storage migration a VM fails to start, reporting;
VM xxx was started by admin (Host: Host-A.x.y).
VM xxx is down. Exit message: ('Failed to get size for volume %s', 'e74e919a-cab1-4d09-8f35-ad72f3c51154').
Resolution
-
A root cause for the problem was not found, as the logs captured did not encompass the time-frames in question.
-
The RHEV database was modified to reflect the current state of the volumes in the storage domain. Please contact Red Hat Technical Support for details of the steps taken,
Root Cause
-
The reason for the Failed to get size for volume error was that the volume in question no longer physically existed in the Storage Domain.
-
In this specific case the following occurred;
- The VM consisted of 4 disks.
- Each disk consisted of 2 images in the RHEV database.
- For the first two disks, both of the images existed in the storage domain as logical volumes. However, the linkage of these was incorrect. The
parentidfields (and the "PU" LVM tags) of the leaf images (the images/LVs created by a snapshot function) pointed to non-existent images/LVs. Also, the base image of each disk was the current active image, i.e. the leaf image was not the active image, as it would normally be. - For the third disk, the base image existed in the storage domain, but the leaf image did not. Again, the
parentidfield and the "PU" LVM tag of the leaf image pointed to a non-existent image/LV and the base image was the current active image. - For the fourth disk, the base image existed in the storage domain, but the leaf image did not. Again, the
parentidfield and the "PU" LVM tag of the leaf image pointed to a non-existent image/LV, but in this case the current active image was the leaf image, not the base image. Since this image did not exist, this is what caused the VM not to start. Also, in the storage domain there was another LV that associated with this disk. Its "PU" LVM tag pointed to the base image of the disk. However, this image was not referenced in the database at all. - The leaf images/LVs for the first, second and third disks were 1gb. This would imply that they were probably newly-created images that hadn't been used yet.
- The leaf image for the fourth disk was 72gb. This would imply that this image held some meaningful data at some point, however this image/:V no longer physically existed.
- The VM had 2 snapshots table entries in the RHEV database (meaning there was one snapshot).
- One snapshots entry was in "PREVIEW" mode.
- Normally in a snapshot preview, there would be another snapshots table entry that would be in an "IN_PREVIEW" state. This did not exist.
- The "PREVIEW" snapshot was associated with the leaf images of the first, second and third disks and the "ACTIVE" snapshots were associated with the base images. However, for the fourth disk, the "PREVIEW" snapshot was associated with the base image and the "ACTIVE" snapshot was associated with the leaf image.
- The third and fourth disks were moved from one storage domain to another a few weeks before.. There seemed to be some issue with the move of the fourh disk. After/during this the VM wouldn't start, indicating that the active image for that disk did not exist (as mentioned above). However, the engine logs didn't indicate any problem occurring during the disk move. Unfortunately the SPM logs did not go back far enough to encompass the time-frame in question.
- Later on the same day that the disks were moved, a snapshot deletion was performed. This seems to have (almost) completed successfully. The
parentidfields and PU tags still pointed to the images that were deleted. - Some time later, a snapshot preview was performed. As mentioned above, this did not complete successfully.
Diagnostic Steps
- From the RHEV database, there were 4 disks;
vm_device:
device_id | vm_id | boot_order | _create_date | _update_date | address
--------------------------------------+--------------------------------------+------------+-------------------------------+-------------------------------+--------------------------------------------------------------
fbf660d0-3140-4c23-a3fc-746bebc834d9 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | 0 | 2014-06-16 14:24:02.969389-04 | 2014-11-28 06:06:16.721534-05 | {bus=0x00, domain=0x0000, type=pci, slot=0x0a, function=0x0}
dedac08d-5f42-4715-a6f9-2bd93c9f3145 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | 1 | 2014-06-16 14:24:02.969389-04 | 2014-11-28 06:06:16.721534-05 | {bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}
1f474a2d-225a-4f50-80e7-f4bf26103956 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | 0 | 2014-06-16 14:24:02.969389-04 | 2014-11-28 06:06:16.721534-05 | {bus=0x00, domain=0x0000, type=pci, slot=0x09, function=0x0}
eb6c81b2-f743-446e-a9b1-66ff6d6b5e33 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | 0 | 2014-06-16 14:24:02.969389-04 | 2014-11-26 08:47:11.952413-05 | {bus=0x00, domain=0x0000, type=pci, slot=0x0b, function=0x0}
(4 rows)
- The images for each disk;
images: (1 = ok, 2 = locked)
image_guid | image_group_id | vm_snapshot_id | parentid | imagestatus | creation_date | volume_type | volume_format | active
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------+------------------------+-------------+---------------+--------
9050a961-a8e9-47bd-957c-aea3002502f7 | fbf660d0-3140-4c23-a3fc-746bebc834d9 | 8b9f8863-7bd6-47ca-b4f8-09b1c27a30b6 | 00000000-0000-0000-0000-000000000000 | 1 | 2014-11-26 04:38:14-05 | 1 | 5 | t
acc7bfc8-2b7f-461c-820d-7381d2408622 | fbf660d0-3140-4c23-a3fc-746bebc834d9 | 57c215c5-3800-4536-93a5-9467105cf653 | 670e4590-3546-4385-93b8-6515a1b1a8f9 | 1 | 2014-06-16 11:33:24-04 | 2 | 4 | f
(2 rows)
For image_guid = 9050a961-a8e9-47bd-957c-aea3002502f7 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 42949672960
For image_guid = acc7bfc8-2b7f-461c-820d-7381d2408622 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 1073741824
image_guid | image_group_id | vm_snapshot_id | parentid | imagestatus | creation_date | volume_type | volume_format | active
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------+------------------------+-------------+---------------+--------
46cba597-488d-4c5c-9c3f-0910d4cd2540 | dedac08d-5f42-4715-a6f9-2bd93c9f3145 | 8b9f8863-7bd6-47ca-b4f8-09b1c27a30b6 | 00000000-0000-0000-0000-000000000000 | 1 | 2014-06-16 14:13:10-04 | 1 | 5 | t
1fb6d6ec-1d8f-48fe-bd87-bd5f0b842769 | dedac08d-5f42-4715-a6f9-2bd93c9f3145 | 57c215c5-3800-4536-93a5-9467105cf653 | caf129cf-2fe8-4e67-8485-1cb24836a9f2 | 1 | 2014-06-16 11:33:28-04 | 2 | 4 | f
(2 rows)
For image_guid = 46cba597-488d-4c5c-9c3f-0910d4cd2540 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 42949672960
For image_guid = 1fb6d6ec-1d8f-48fe-bd87-bd5f0b842769 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 1073741824
image_guid | image_group_id | vm_snapshot_id | parentid | imagestatus | creation_date | volume_type | volume_format | active
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------+------------------------+-------------+---------------+--------
8a2f7142-9785-4957-bc86-b2902b9beda2 | 1f474a2d-225a-4f50-80e7-f4bf26103956 | 8b9f8863-7bd6-47ca-b4f8-09b1c27a30b6 | 00000000-0000-0000-0000-000000000000 | 1 | 2014-06-16 14:13:12-04 | 1 | 5 | t
aecbc6bb-7f61-4140-929b-18268c23603d | 1f474a2d-225a-4f50-80e7-f4bf26103956 | 57c215c5-3800-4536-93a5-9467105cf653 | 5a73866a-8ec4-4dbe-8016-ed7402715bf9 | 1 | 2014-06-16 11:33:26-04 | 2 | 4 | f
(2 rows)
For image_guid = 8a2f7142-9785-4957-bc86-b2902b9beda2 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 42949672960
For image_guid = aecbc6bb-7f61-4140-929b-18268c23603d , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 1073741824
image_guid | image_group_id | vm_snapshot_id | parentid | imagestatus | creation_date | volume_type | volume_format | active
--------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+-------------+------------------------+-------------+---------------+--------
d9d6bae1-765d-478b-a2da-ff5e314a0e58 | eb6c81b2-f743-446e-a9b1-66ff6d6b5e33 | 57c215c5-3800-4536-93a5-9467105cf653 | 00000000-0000-0000-0000-000000000000 | 1 | 2014-06-16 11:33:29-04 | 1 | 5 | f
e74e919a-cab1-4d09-8f35-ad72f3c51154 | eb6c81b2-f743-446e-a9b1-66ff6d6b5e33 | 8b9f8863-7bd6-47ca-b4f8-09b1c27a30b6 | 3c8347bc-d0bc-4d42-950a-904065491bb0 | 1 | 2014-06-16 14:23:49-04 | 2 | 4 | t
(2 rows)
For image_guid = d9d6bae1-765d-478b-a2da-ff5e314a0e58 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 107374182400
For image_guid = e74e919a-cab1-4d09-8f35-ad72f3c51154 , SD = 529e1ae5-74b2-4c13-b040-d54cddb19957 ; image size = 77309411328
- Snapshots;
snapshots:
snapshot_id | vm_id | snapshot_type | status | description | creation_date
--------------------------------------+--------------------------------------+---------------+--------+------------------------------+----------------------------
57c215c5-3800-4536-93a5-9467105cf653 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | PREVIEW | OK | Active VM before the preview | 2014-06-16 14:13:12.94-04
8b9f8863-7bd6-47ca-b4f8-09b1c27a30b6 | 0ed2eee8-1bcc-46ea-951d-614414815d9e | ACTIVE | OK | Active VM | 2014-12-05 04:44:47.934-05
(2 rows)
- Volumes in the storage domain;
- LVs;
$ grep fbf660d0-3140-4c23-a3fc-746bebc834d9 lvm_lvs_-v_-o_tags
9050a961-a8e9-47bd-957c-aea3002502f7 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 40.00g -1 -1 -1 -1 s1eEhM-fw6t-Shng-qZqg-Cgxg-R0fH-IX48CE PU_00000000-0000-0000-0000-000000000000,IU_fbf660d0-3140-4c23-a3fc-746bebc834d9,MD_38
$ grep dedac08d-5f42-4715-a6f9-2bd93c9f3145 lvm_lvs_-v_-o_tags
1fb6d6ec-1d8f-48fe-bd87-bd5f0b842769 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 1.00g -1 -1 -1 -1 5b5f27-aAxo-0IGj-Kzz3-g5I8-zixg-ZKNhp4 MD_20,IU_dedac08d-5f42-4715-a6f9-2bd93c9f3145,PU_caf129cf-2fe8-4e67-8485-1cb24836a9f2
46cba597-488d-4c5c-9c3f-0910d4cd2540 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 40.00g -1 -1 -1 -1 LPur0P-69z5-CbJ7-AsnF-h8aU-lDw1-46fs6f MD_36,PU_00000000-0000-0000-0000-000000000000,IU_dedac08d-5f42-4715-a6f9-2bd93c9f3145
$ grep 1f474a2d-225a-4f50-80e7-f4bf26103956 lvm_lvs_-v_-o_tags
8a2f7142-9785-4957-bc86-b2902b9beda2 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 40.00g -1 -1 -1 -1 ikpcLy-WpDC-D5XZ-IVgZ-OUIT-RBzv-en1WVt IU_1f474a2d-225a-4f50-80e7-f4bf26103956,MD_37,PU_00000000-0000-0000-0000-000000000000
aecbc6bb-7f61-4140-929b-18268c23603d 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 1.00g -1 -1 -1 -1 LIITNd-x4eb-XUaP-nd9Y-YabD-Okmt-FBfTsy IU_1f474a2d-225a-4f50-80e7-f4bf26103956,PU_5a73866a-8ec4-4dbe-8016-ed7402715bf9,MD_19
$ grep eb6c81b2-f743-446e-a9b1-66ff6d6b5e33 lvm_lvs_-v_-o_tags
30a6b3d1-bb87-41f6-ae0c-f2c0192b1eab 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 1.00g -1 -1 -1 -1 Fcfv0L-eH56-rRS0-eBjV-CcR8-3v2n-YLBpTI PU_d9d6bae1-765d-478b-a2da-ff5e314a0e58,IU_eb6c81b2-f743-446e-a9b1-66ff6d6b5e33,MD_54
d9d6bae1-765d-478b-a2da-ff5e314a0e58 529e1ae5-74b2-4c13-b040-d54cddb19957 1 -wi------- 100.00g -1 -1 -1 -1 kG2zI5-OeQm-5xpt-gaYf-4GxF-nvU8-M3Eokc IU_eb6c81b2-f743-446e-a9b1-66ff6d6b5e33,PU_00000000-0000-0000-0000-000000000000,MD_39
$ grep e74e919a-cab1-4d09-8f35-ad72f3c51154 lvm_lvs_-v_-o_tags
$
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
