Instance creation randomly fails with "Failed to allocate the network(s), not rescheduling" due to flapping SR-IOV ports exhausting the Neutron event queue

Issue

• Instance creation fails intermittently and randomly due to Failed to allocate the network(s), not rescheduling error.

  </var/log/containers/nova/nova-compute.log on a compute node>
  WARNING nova.virt.libvirt.driver [...] [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa] Timeout waiting for [('network-vif-plugged', 'bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb')] for instance with vm_state building and task_state spawning: eventlet.timeout.Timeout: 300 seconds
  ERROR nova.virt.block_device [...] [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa] Build of instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa aborted: Failed to allocate the network(s), not rescheduling.: nova.exception.BuildAbortException: Build of instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa aborted: Failed to allocate the network(s), not rescheduling.
  

• neutron-server logs show an abnormally high rate of OVN reports status up/down for port: <port-id> messages (e.g.: tens of thousands per hour).

  </var/log/containers/nova/nova-compute.log on a controller node>
    :
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status up for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status up for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status up for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [...] OVN reports status down for port: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa