VM creation stopped working after running openstack overcloud deploy

Issue

• We can't create new instance inside of a Openstack cluster after running an "openstack overcloud deploy" as we see errors similar to this in nova-compute.log:

Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed

• Do you think upgrading to 16.2.5 would fix the issue?

• We found the below message in openvswitch logs /var/log/containers/openvswitch/ovn-controller.log.4.gz:

2023-08-17T05:11:49.781Z|03423|main|WARN|controller version - 21.12.3-20.21.0-62.4 mismatch with northd version - 21.12.3-20.21.0-61.4

• openstack network agent list shows all agents appears to no longer be alive:

| ID                                   | Agent Type                   | Host                      | Availability Zone | Alive | State | Binary                     |
+--------------------------------------+------------------------------+---------------------------+-------------------+-------+-------+----------------------------+
| bc58a243-c9e1-48bc-aa25-23bc6a18e956 | OVN Controller agent         | compute-0.redhat.local    |                   | XXX   | UP    | ovn-controller             |
| 32294417-48c4-507a-b7cd-5affaa62e009 | OVN Metadata agent           | compute-0.redhat.local    |                   | XXX   | UP    | neutron-ovn-metadata-agent |
| de3b2308-8cf1-4453-80a5-5bb7898c084f | OVN Controller Gateway agent | controller-0.redhat.local |                   | XXX   | UP    | ovn-controller             |
| af5eb2d5-d51e-40f7-9dab-04d31588c034 | OVN Controller Gateway agent | controller-1.redhat.local |                   | XXX   | UP    | ovn-controller             |
| 985726c7-09f8-45f7-bfe8-9e34cde5fe30 | OVN Controller Gateway agent | controller-2.redhat.local |                   | XXX   | UP    | ovn-controller             |