After upgrading from 16.2 to 17.1 instances fail to start with version negotiation failed (we support version 0x06, peer supports versions 0x01, 0x04, 0x05)

Issue

• After upgrading from 16.2 to 17.1 instances are not able to start.
• During the upgrade the neutron-ovs.yaml template was includes and the environment changes from OVN to OVS, but be corrected it back

This error can be seen in nova-compute.log

2024-08-29 09:26:09.502 2 WARNING nova.virt.libvirt.driver [req-e0c9749f-81ec-42b3-901b-391e193f715b b9b8d965b2774c28b175b59aeafb9761 0c030b8d5ad548d7afc2f46844e159d8 - default default] [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] Timeout waiting for [('network-vif-plugged', 'e7cd599d-2121-4b38-a6f6-bfe8c10e4eb0')] for instance with vm_state building and task_state spawning: eventlet.timeout.Timeout: 300 seconds
..
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [req-e0c9749f-81ec-42b3-901b-391e193f715b b9b8d965b2774c28b175b59aeafb9761 0c030b8d5ad548d7afc2f46844e159d8 - default default] [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] Traceback (most recent call last):
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7380, in _create_guest_with_network
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     guest = self._create_guest(
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib64/python3.9/contextlib.py", line 126, in __exit__
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     next(self.gen)
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 481, in wait_for_instance_event
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     actual_event = event.wait()
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/eventlet/event.py", line 125, in wait
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     result = hub.switch()
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     return self.greenlet.switch()
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] eventlet.timeout.Timeout: 300 seconds
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] 
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] During handling of the above exception, another exception occurred:
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] 
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] Traceback (most recent call last):
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2749, in _build_resources
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     yield resources
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2507, in _build_and_run_instance
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     self.driver.spawn(context, instance, image_meta,
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4288, in spawn
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     self._create_guest_with_network(
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7406, in _create_guest_with_network
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e]     raise exception.VirtualInterfaceCreateException()
2024-08-29 09:26:10.721 2 ERROR nova.compute.manager [instance: 4640fb02-55ad-4c42-932f-53a6101f648e] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed

An this in the compute ovn-controller logs

2024-08-22T18:01:41.338Z|515710|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)
2024-08-22T18:01:49.348Z|515712|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x06, peer supports versions 0x01, 0x04, 0x05)        
2024-08-22T18:01:49.348Z|515713|rconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: connection dropped (Broken pipe)