Nova evacuate fails

Issue

• Performing a nova evacuation on an instance with an attached volume fails with:

'ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)'

• Traceback of failure in nova-compute.log:

nova-compute.log:2016-03-10 11:20:23.867 3316 ERROR nova.compute.manager [req-<uuid> - - -] [instance: <uuid>] Setting instance vm_state to ERROR
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] Traceback (most recent call last):
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6476, in _error_out_instance_on_exception
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     yield
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3113, in rebuild_instance
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     self._rebuild_default_impl(**kwargs)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2942, in _rebuild_default_impl
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     detach_block_devices(context, bdms)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3091, in detach_block_devices 
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     destroy_bdm=False)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4956, in _detach_volume
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     self.volume_api.terminate_connection(context, volume_id, connector)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     res = method(self, ctx, volume_id, *args, **kwargs)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     connector)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     {'connector': connector})     
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     return self.api.client.post(url, body=body)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     return self._cs_request(url, 'POST', **kwargs)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     return self.request(url, method, **kwargs)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]     raise exceptions.from_response(resp, body)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-<uuid>)
nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>]

• The below error is seen in the cinder api log:

2016-03-10 11:20:23.863 5316 ERROR cinder.api.middleware.fault [req-<uuid> - - -] Caught error: Remote error: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume <uuid>.
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1181, in terminate_connection\n    raise exception.VolumeBackendAPIException(data=err_msg)\n', u'VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume <uuid>.\n']. None
None.
2016-03-10 11:20:23.864 5316 INFO cinder.api.middleware.fault [req-<uuid> - - -] http://<ip>:8776/v2/<uuid>/volumes/<uuid>/action returned with HTTP 500