Heat stack update fails with port still in use error when user-data changes

Issue

We are trying to update a heat stack ( not the overcloud ) and it fails with "libvirtError: Unable to create tap device tapd68d6cbf-76: Device or resource busy". Special in this heat stack is that we change the image and the config-drive during the update. With the image alone it works, but with the config-drive it fails.

2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager [req-8bf6b43a-8c43-44fb-ac87-1c4a338765fa b14a34d6054f460a99bcd0e9ef0b1fd8 5b192cee71134faa9259906345ad2926 - - -] Instance failed network setup after 1 attempt(s)
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager Traceback (most recent call last):
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1399, in _allocate_network_async
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 840, in allocate_for_instance
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager     context, instance, neutron, requested_networks)
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 558, in _validate_requested_port_ids
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager     raise exception.PortInUse(port_id=request.port_id)
2017-06-01 07:56:05.764 2589 ERROR nova.compute.manager PortInUse: Port d68d6cbf-76c3-4019-acad-79c15dfb4a5e is still in use.
...
2017-06-01 07:56:08.042 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852] PortInUse: Port d68d6cbf-76c3-4019-acad-79c15dfb4a5e is still in use.
...
2017-06-01 07:56:22.638 2589 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2da513e131fa4b31a27ff6ef3f42d05f __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:346
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [req-8bf6b43a-8c43-44fb-ac87-1c4a338765fa b14a34d6054f460a99bcd0e9ef0b1fd8 5b192cee71134faa9259906345ad2926 - - -] [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852] Instance failed to spawn
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852] Traceback (most recent call last):
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     yield resources
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     block_device_info=block_device_info)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2698, in spawn
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     destroy_disks_on_failure=True)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5112, in _create_domain_and_network
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     destroy_disks_on_failure)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     self.force_reraise()
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     six.reraise(self.type_, self.value, self.tb)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5084, in _create_domain_and_network
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     post_xml_callback=post_xml_callback)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5002, in _create_domain
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     guest.launch(pause=pause)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     self._encoded_xml, errors='ignore')
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     self.force_reraise()
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     six.reraise(self.type_, self.value, self.tb)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     return self._domain.createWithFlags(flags)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     rv = execute(f, *args, **kwargs)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     six.reraise(c, e, tb)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     rv = meth(*args, **kwargs)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-06-01 07:56:22.650 2589 ERROR nova.compute.manager [instance: 5b8b0627-82d2-44cf-be8c-c602a4b0c852] libvirtError: Unable to create tap device tapd68d6cbf-76: Device or resource busy