How to migrate in-use cinder boot volumes from one cinder backend to another in Red Hat OpenStack Platform

Issue

How to migrate in-use cinder boot volumes from one cinder backend to another in Red Hat OpenStack Platform?

The following test does not work:

nova flavor-create m1.small.tiny.disk auto 1024 10 1
openstack volume create --size 10 --image 4da90396-4017-4627-9d04-1f20af9a1409 bfv3
nova boot bfv3 --flavor m1.small.tiny.disk --nic net-id=$NETID --block-device source=volume,id=db589471-b72c-4ab7-bf9a-46c7c407afd6,dest=volume,size=10,shutdown=preserve,bootindex=0 --key-name id_rsa

The volume is currently on nfs1

[stack@undercloud-1 ~]$ cinder show db589471-b72c-4ab7-bf9a-46c7c407afd6 | grep nfs
| os-vol-host-attr:host          | hostgroup@nfs1#nfs1 

Retyping to nfs2:

cinder retype --migration-policy on-demand db589471-b72c-4ab7-bf9a-46c7c407afd6 nfs2

This leads to:

[root@overcloud-controller-0 ~]# tail -f /var/log/cinder/volume.log | egrep 'TRACE|ERROR'
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager [req-f85f3996-61c9-4627-9b03-80e38918981d 62b9e2a5841942bdabb43e2b4be43df8 db2b3d12c0c644a19780132399321045 - default default] Failed to copy volume db589471-b72c-4ab7-bf9a-46c7c407afd6 to 2f28de39-7bc9-4ce9-98e6-c2e1a0d92f9b
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager Traceback (most recent call last):
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1796, in _migrate_volume_generic
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     new_volume.id)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/compute/nova.py", line 181, in update_server_volume
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     new_volume_id)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 68, in update_server_volume
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     body, "volumeAttachment")
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 370, in _update
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     resp, body = self.api.client.put(url, body=body)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 196, in put
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     return self.request(url, 'PUT', **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 168, in request
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 344, in request
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 112, in request
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     return self.session.request(url, method, **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     return wrapped(*args, **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 486, in request
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     auth_headers = self.get_auth_headers(auth)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 757, in get_auth_headers
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     return auth.get_headers(self, **kwargs)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     token = self.get_token(session)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     return self.get_access(session).auth_token
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     self.auth_ref = self.get_auth_ref(session)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     self._plugin = self._do_create_plugin(session)
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 174, in _do_create_plugin
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager     raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
2017-09-22 18:36:08.579 20858 ERROR cinder.volume.manager DiscoveryFailure: Could not determine a suitable URL for the plugin
(...)

A solution could be to detach the volume and then migrate it. Unfortunately, the volume cannot be simply detached - it is the root volume of a boot from volume device:

[stack@undercloud-1 ~]$ nova stop bfv3
Request to stop server bfv3 has been accepted.
[stack@undercloud-1 ~]$ nova volume-detach bfv3 db589471-b72c-4ab7-bf9a-46c7c407afd6
ERROR (Forbidden): Can't detach root device volume (HTTP 403) (Request-ID: req-7e9eb1f6-2100-49ca-956f-c02a871d82c7)
[stack@undercloud-1 ~]$