Unable to attach or boot from rbd (Ceph) Cinder volumes

Issue

• Unable to boot instances from rbd volumes.

• Unable to attach rbd volumes to instances.

• The following errors seen in /var/log/libvirt/libvirtd.log when attempting to attach rbd volumes to instances :

2014-06-13 19:03:24.479+0000: 2675: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized
2014-06-23 13:07:58.052+0000: 2674: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized
2014-06-23 13:08:43.336+0000: 2678: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized
2014-06-23 13:09:25.826+0000: 2675: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized
2014-06-23 13:10:24.077+0000: 2677: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized
2014-06-23 14:29:12.644+0000: 2676: error : qemuMonitorJSONCheckError:357 : internal error unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk3' could not be initialized

• The following trace seen in /var/log/nova/compute.log when attempting to boot an instance from a rbd volume :

2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a] Traceback (most recent call last):
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1045, in _build_instance
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     set_access_ip=set_access_ip)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1428, in _spawn
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1425, in _spawn
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     block_device_info)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2091, in spawn
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     block_device_info, context=context)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3249, in _create_domain_and_network
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     domain = self._create_domain(xml, instance=instance, power_on=power_on)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3192, in _create_domain
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     domain.XMLDesc(0))
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3187, in _create_domain
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     domain.createWithFlags(launch_flags)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in proxy_call
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     rv = execute(f,*args,**kwargs)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     rv = meth(*args,**kwargs)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in createWithFlags
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a] libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/3
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a] qemu-kvm: -drive file=rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=bf74268a-e503-460e-9c31-419cbbef7687,cache=none: error loading librbd: /usr/lib64/qemu/librbd.so.1: cannot open shared object file: No such file or directory
2014-06-13 14:38:55.882 1589 TRACE nova.compute.manager [instance: 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a] qemu-kvm: -drive file=rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=bf74268a-e503-460e-9c31-419cbbef7687,cache=none: could not open disk image rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1

• The following error is seen in /var/log/libvirt/qemu/${instance_name}.log when attempting to boot an instance from a rbd volume :

2014-06-13 18:38:51.575+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00000004 -S -M rhel6.5.0 -no-kvm -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=2013.2.3-7.el6ost,serial=c7b0d959-809b-42d9-8841-d2accb92d600,uuid=9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000004.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=bf74268a-e503-460e-9c31-419cbbef7687,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c3:dd:11,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/9c73ddf9-fb8a-40d1-ab6d-901385ea2b0a/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.33.20.129:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/3
qemu-kvm: -drive file=rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=bf74268a-e503-460e-9c31-419cbbef7687,cache=none: error loading librbd: /usr/lib64/qemu/librbd.so.1: cannot open shared object file: No such file or directory
qemu-kvm: -drive file=rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=bf74268a-e503-460e-9c31-419cbbef7687,cache=none: could not open disk image rbd:volumes/volume-bf74268a-e503-460e-9c31-419cbbef7687:id=cinder:key=AQBtCJtTiH2rJhAATVdfuz8jbOO5ywljbKC59g==:auth_supported=cephx\;none:mon_host=192.1.0.3\:6789\;192.1.0.4\:6789\;192.1.0.5\:6789: Input/output error
2014-06-13 18:38:51.735+0000: shutting down