RHEV: A Virtual Machine with a large number of snapshots may fail to start and/or no new snapshots can be created.

Solution In Progress - Updated -

Issue

  • A VM fails to start and reports the following in the RHEV Admin Portal;
VM <vm-name> is down. Exit message: Unable to read from monitor: Connection reset by peer.
  • The vdsm logs on the host that the VM tried to start on contain;
Thread-3743::DEBUG::2014-02-13 10:03:45,861::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 10 level: 2 message: Unable to read from monitor: Connection reset by peer
  • The qemu log for the VM in question contains;
-drive file=/rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2,if=none,id=drive-virtio-disk0,format=qcow2,serial=333a1d04-5b14-479a-8716-c753f7e539eb,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/mnt/blockSD/688056fa-6477-47c0-8da8-a99b068d942d/images/333a1d04-5b14-479a-8716-c753f7e539eb/9c3b54da-95aa-4c34-aaa6-5651c06baab2: No such file or directory
  • Attempting to create a new snapshot fails with the following sequence reported in the vdsm logs;
b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,142::volume::1058::Storage.Misc.excCmd::(createVolume) '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b ../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7d
fe3fb6 /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27' (cwd /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d83
1b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920)

......... etc., etc., .........

b846ac80-2ccb-4d11-b5af-3301d8330bb3::DEBUG::2014-02-20 10:06:28,291::volume::1058::Storage.Misc.excCmd::(createVolume) FAILED: <err> = '/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: error while creating qcow2: No such file or directory\n'; <rc> = 1

......... etc., etc., .........

b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::286::Storage.Volume::(clone) Volume.clone: can't clone: /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6 to /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27
b846ac80-2ccb-4d11-b5af-3301d8330bb3::ERROR::2014-02-20 10:06:37,408::volume::508::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/volume.py", line 466, in create
    srcVolUUID, imgPath, volPath)
  File "/usr/share/vdsm/storage/blockVolume.py", line 197, in _create
    volParent.clone(imgPath, volUUID, volFormat, preallocate)
  File "/usr/share/vdsm/storage/volume.py", line 287, in clone
    raise se.CannotCloneVolume(self.volumePath, dst_path, str(e))
CannotCloneVolume: Cannot clone volume: 'src=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6, dst=/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27: Error creating a new volume: (["Formatting \'/rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/6ae24343-d09c-45e8-bfbf-fc1aee023ea5/images/29612aa0-dc06-453e-9fcc-2cdb63d2d920/3bb487ef-3809-4a0c-817c-68da9d7c2b27\', fmt=qcow2 size=1073741824 backing_file=\'../29612aa0-dc06-453e-9fcc-2cdb63d2d920/3ac09418-9f1b-431b-83b8-7f9b7dfe3fb6\' backing_fmt=\'qcow2\' encryption=off cluster_size=65536 "],)'
  • After upgrading a RHEV host to vdsm-4.13.2-0.6 and qemu-kvm-rhev-0.12.1.2-2.415 or later versions, a VM with more than 23 snapshots failed to start. Attempting to create a new snapshot also failed.

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.2, 3.3
  • qemu-kvm-rhev-0.12.1.2-2.415 or later

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.