Iscsiadm reporting login errors on storage for volumes deleted a long time ago

Issue

• iscsiadm reporting login errors on storage for volumes deleted a long time ago such as this:

Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-1f42ae69-7046-46c1-ad60-02c926d57402.641, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-31fd3842-e488-4aaf-aae7-40dad3a571f1.713, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-3865c73a-0d42-43f7-b821-48bc31cf9c76.718, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-916e0d77-a71e-495b-9236-1fa9fdcb890b.721, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-937ae34d-5c13-41f5-8fc2-7b063159a053.749, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-7a5c337b-4cab-4435-9420-cd3f4654c752.773, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-d468a9e2-bddb-4080-a55e-16aef5019953.774, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-f6c77fe3-9b79-43e4-9391-f708b0c5397a.775, portal: 10.10.10.10,3260].
Apr 15 14:35:44 overcloud-compute-0 iscsiadm: iscsiadm: Could not login to [iface: default, target: iqn.2010-01.com.solidfire:pbpl.uuid-1ddeab22-5913-4356-86e9-60451a73d911.806, portal: 10.10.10.10,3260].

• openstack-nova-compute occasionnaly fails to delete an attached volume :

2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server [req-ff4ec5df-4fcd-440f-9e24-1410007000b6 b7e901ba6e8d4c8e94a1f9cb1d85741d 93e2d810a0354cb4b821839a6ae33b54 - - -] Exception during message handling
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4871, in detach_volume
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4806, in _detach_volume
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self._driver_detach_volume(context, instance, bdm, connection_info)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4755, in _driver_detach_volume
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.volume_api.roll_detaching(context, volume_id)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4743, in _driver_detach_volume
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     encryption=encryption)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1368, in detach_volume
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     wait_for_detach()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 385, in func
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return evt.wait()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return hubs.get_hub().switch()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 136, in _run_loop
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     result = func(*self.args, **self.kw)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 356, in _func
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 445, in _do_wait_and_retry_detach
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     _try_detach_device(config, persistent=False, live=live)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 409, in _try_detach_device
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     device=alternative_device_name)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 389, in _try_detach_device
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self.detach_device(conf, persistent=persistent, live=live)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 466, in detach_device
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     self._domain.detachDeviceFlags(device_xml, flags=flags)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1251, in detachDeviceFlags
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2019-03-28 19:25:52.745 622693 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk4' not found