OSP10: Instance instantiation failing with 'aborted: Failed to allocate the network(s), not rescheduling., Code: 500'

Issue

• new instances are Erroring with the following fault output:

  {u'message': u'Build of instance 0b4b5528-1440-4086-9b16-eb49e357efc0 aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'details': u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1787, in _do_build_and_run_instance\n    filter_properties)\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1968, in _build_and_run_instance\n    reason=msg)\n', u'created': u'2019-10-08T18:02:10Z'}
  

• this is happening across multiple projects and networks.

  [root@controller1 log]# grep 'Failed to allocate the network' /var/log/*/*.log
  /var/log/heat/heat-engine.log:2019-09-24 21:07:33.638 425918 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: Build of instance 19145035-3344-4f2a-9fbf-583a446084b7 aborted: Failed to allocate the network(s), not rescheduling., Code: 500"
  /var/log/heat/heat-engine.log:2019-09-24 21:13:02.468 425918 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: Build of instance 0fa17c2a-4a7e-4865-9209-4c2ac6d603ca aborted: Failed to allocate the network(s), not rescheduling., Code: 500"
  /var/log/heat/heat-engine.log:2019-09-24 21:18:33.774 425918 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: Build of instance 8ca12f90-ee8a-4b04-8b61-3fcb5fd6e1b8 aborted: Failed to allocate the network(s), not rescheduling., Code: 500"
  /var/log/heat/heat-engine.log:2019-09-24 21:24:08.939 425918 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: Build of instance 95c3c8ee-f11d-45ae-b485-d8e85bd9a0d0 aborted: Failed to allocate the network(s), not rescheduling., Code: 500"