Attach RBD volume on VM using different nova backend RBD

Issue

• We have two ceph clusters configured on our environment.

• On an existing CEPH cluster, the following pools cinder_sata, cinder_ssd, nova and glance we created and everything is working as expected.

• On a new CEPH cluster, a new pool was created for cinder (cinder_new) and creating volumes on this pool works as expected.

• When a volume is created on the new cluster and then attached to a VM, it fails as the secret UUID from the existing cluster is used to attach the volume from the new ceph cluster.

• The following error message is seen in /var/log/nova/nova-compute.log on the compute where the VM is hosted:

2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [req-865d6e73-5b05-4ddd-b652-b3c6624526f4 02887d1caa5e47b3b636a6f226baa7e5 b6368111ff0241fd8d07bdf82e308b3d - - -] [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9] Failed to attach volume at mountpoint: /dev/sdb
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9] Traceback (most recent call last):
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1176, in attach_volume
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     guest.attach_device(conf, persistent=True, live=live)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 305, in attach_device
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     rv = execute(f, *args, **kwargs)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     six.reraise(c, e, tb)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     rv = meth(*args, **kwargs)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 564, in attachDeviceFlags
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9] libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi0-0-0-1' could not be initialized
2019-04-15 19:57:33.683 958735 ERROR nova.virt.libvirt.driver [instance: 53b645ed-bda9-451a-bc20-feb7dd4a13a9]