Cinder does not retype volume with Hitachi backend

Issue

• Why cinder does not retype volume with Hitachi backend?

• Configured multiple Hitachi backends differentiated by port assigment. Cinder retype is not possible from backend to another.

enabled_backends = Tier1, Tier2, Tier3
default_volume_type = Tier3

[Tier1]
volume_driver=cinder.volume.drivers.hitachi.hbsd.hbsd_fc.HBSDFCDriver
volume_backend_name=Tier1
hitachi_storage_id=xxxxx
hitachi_pool=005
hitachi_thin_pool=008
hitachi_horcm_user=OS
hitachi_horcm_password_path=/opt/hitachi/hbsd/horcm_password
hitachi_target_ports=CL1-F,CL2-F
hitachi_compute_target_ports=CL1-F,CL2-F
hitachi_zoning_request=False
hitachi_default_copy_method=FULL
use_multipath_for_image_xfer=True
hitachi_copy_version=1.0 
hitachi_horcm_add_conf=False
hitachi_horcm_numbers=200,201

[Tier2]
volume_driver=cinder.volume.drivers.hitachi.hbsd.hbsd_fc.HBSDFCDriver
volume_backend_name=Tier2
hitachi_storage_id=xxxxx
hitachi_pool=006
hitachi_thin_pool=008
hitachi_horcm_user=OS
hitachi_horcm_password_path=/opt/hitachi/hbsd/horcm_password
hitachi_target_ports=CL1-F,CL2-F
hitachi_compute_target_ports=CL1-F,CL2-F
hitachi_zoning_request=False
hitachi_default_copy_method=FULL
use_multipath_for_image_xfer=True
hitachi_copy_version=1.0 
hitachi_horcm_add_conf=False
hitachi_horcm_numbers=202,203

[Tier3]
volume_driver=cinder.volume.drivers.hitachi.hbsd.hbsd_fc.HBSDFCDriver
volume_backend_name=Tier3
hitachi_storage_id=xxxxx
hitachi_pool=007
hitachi_thin_pool=008
hitachi_horcm_user=OS
hitachi_horcm_password_path=/opt/hitachi/hbsd/horcm_password
hitachi_target_ports=CL1-F,CL2-F
hitachi_compute_target_ports=CL1-F,CL2-F
hitachi_zoning_request=False
hitachi_default_copy_method=FULL
use_multipath_for_image_xfer=True
hitachi_copy_version=1.0 
hitachi_horcm_add_conf=False
hitachi_horcm_numbers=204,205

• cinder retype operation is getting failed with following call trace and it's deleting the temporary cinder volumes which was created during the retype operation.

2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager [req-ce0fc17b-0c9d-46e9-82b3-722de16ca19a 8c1aa3f10dfb47f2b061c3674be85515 eb67a97587234f7790070d0a58032ba7 - - -] Failed to copy volume b949b243-123c-4c0f-80a1-01672966cc10 to 4f2b48c5-8330-450b-b668-6b67103ca238
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager Traceback (most recent call last):
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1768, in _migrate_volume_generic
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1708, in _copy_volume_data
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     remote=dest_remote)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1649, in _detach_volume
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     rpcapi.terminate_connection(ctxt, volume, properties, force=force)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 248, in terminate_connection
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     connector=connector, force=force)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     retry=self.retry)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     timeout=timeout, retry=retry)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     retry=retry)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     result = self._waiter.wait(msg_id, timeout)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     message = self.waiters.get(msg_id, timeout=timeout)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager     'to message ID %s' % msg_id)
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager MessagingTimeout: Timed out waiting for a reply to message ID 69113807c44e41e6914d01acb99d40b6
2017-03-20 13:15:07.884 4435 ERROR cinder.volume.manager 
2017-03-20 13:15:08.070 4436 INFO cinder.volume.drivers.hitachi.hbsd.hbsd_traceutils [req-ce0fc17b-0c9d-46e9-82b3-722de16ca19a 8c1aa3f10dfb47f2b061c3674be85515 eb67a97587234f7790070d0a58032ba7 - - -] MSGID0000-I: The method delete_volume was called. (config_group: Tier2)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher [req-ce0fc17b-0c9d-46e9-82b3-722de16ca19a 8c1aa3f10dfb47f2b061c3674be85515 eb67a97587234f7790070d0a58032ba7 - - -] Exception during message handling: Timed out waiting for a reply to message ID 69113807c44e41e6914d01acb99d40b6
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2310, in retype
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     new_reservations, status_update)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2306, in retype
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     new_type_id=new_type_id)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1983, in migrate_volume
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     volume.save()
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1976, in migrate_volume
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     new_type_id)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1791, in _migrate_volume_generic
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     new_volume)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1768, in _migrate_volume_generic
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     self._copy_volume_data(ctxt, volume, new_volume, remote='dest')
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1708, in _copy_volume_data
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     remote=dest_remote)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1649, in _detach_volume
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     rpcapi.terminate_connection(ctxt, volume, properties, force=force)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 248, in terminate_connection
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     connector=connector, force=force)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     retry=self.retry)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     retry=retry)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     result = self._waiter.wait(msg_id, timeout)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     message = self.waiters.get(msg_id, timeout=timeout)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher     'to message ID %s' % msg_id)
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher MessagingTimeout: Timed out waiting for a reply to message ID 69113807c44e41e6914d01acb99d40b6
2017-03-20 13:15:08.103 4435 ERROR oslo_messaging.rpc.dispatcher 
2017-03-20 13:15:12.464 4436 INFO cinder.volume.drivers.hitachi.hbsd.hbsd_traceutils [req-ce0fc17b-0c9d-46e9-82b3-722de16ca19a 8c1aa3f10dfb47f2b061c3674be85515 eb67a97587234f7790070d0a58032ba7 - - -] MSGID0002-I: The method delete_volume completed successfully. (config_group: Tier2)