RHEV: A Virtual Machine with a large number of snapshots may fail to start and/or no new snapshots can be created.
Issue
- A VM fails to start and reports the following in the RHEV Admin Portal;
VM <vm-name> is down. Exit message: Unable to read from monitor: Connection reset by peer.
- The
vdsmlogs on the host that the VM tried to start on contain;
Thread-3743::DEBUG::2014-02-13 10:03:45,861::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 10 level: 2 message: Unable to read from monitor: Connection reset by peer
- The qemu log for the VM in question contains;
-drive file=/rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2,if=none,id=drive-virtio-disk0,format=qcow2,serial=333a1d04-5b14-479a-8716-c753f7e539eb,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2: No such file or directory
- Attempting to create a new snapshot fails with the following sequence reported in the
vdsmlogs;
b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,142::volume::1058::Storage.Misc.excCmd::(createVolume) '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b ../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7d
fe3fb6 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27' (cwd /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d83
1b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920)
......... etc., etc., .........
b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,291::volume::1058::Storage.Misc.excCmd::(createVolume) FAILED: <err> = '/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: error while creating qcow2: No such file or directory\n'; <rc> = 1
......... etc., etc., .........
b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::286::Storage.Volume::(clone) Volume.clone: can't clone: /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6 to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27
b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::508::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/volume.py", line 466, in create
srcVolUUID, imgPath, volPath)
File "/usr/share/vdsm/storage/blockVolume.py", line 197, in _create
volParent.clone(imgPath, volUUID, volFormat, preallocate)
File "/usr/share/vdsm/storage/volume.py", line 287, in clone
raise se.CannotCloneVolume(self.volumePath, dst_path, str(e))
CannotCloneVolume: Cannot clone volume: 'src=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6, dst=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: Error creating a new volume: (["Formatting \'/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27\', fmt=qcow2 size=1073741824 backing_file=\'../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6\' backing_fmt=\'qcow2\' encryption=off cluster_size=65536 "],)'
- After upgrading a RHEV host to
vdsm-4.13.2-0.6andqemu-kvm-rhev-0.12.1.2-2.415or later versions, a VM with more than 23 snapshots failed to start. Attempting to create a new snapshot also failed.
Environment
- Red Hat Enterprise Virtualization (RHEV) 3.2, 3.3
- qemu-kvm-rhev-0.12.1.2-2.415 or later
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
