Unable to launch instances after a while - Failed to allocate the network(s)

Solution In Progress - Updated -

Issue

  • After having created a lot of VMs, new VMs can no longer be created an the following error message will be seen in openstack server show:
[stack@undercloud ~]$ openstack server show 6014d6fc-dff7-493c-96ac-c01dee94a8d6
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                               | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                   | AUTO                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| OS-EXT-AZ:availability_zone         |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| OS-EXT-SRV-ATTR:host                | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000629                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| OS-EXT-STS:power_state              | NOSTATE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| OS-EXT-STS:task_state               | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| OS-EXT-STS:vm_state                 | error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| OS-SRV-USG:launched_at              | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| OS-SRV-USG:terminated_at            | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| accessIPv4                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| accessIPv6                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| addresses                           |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| config_drive                        |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| created                             | 2019-07-17T19:10:05Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| fault                               | {u'message': u'Build of instance 6014d6fc-dff7-493c-96ac-c01dee94a8d6 aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'details': u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1858, in _do_build_and_run_instance\n    filter_properties, request_spec)\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2113, in _build_and_run_instance\n    reason=msg)\n', u'created': u'2019-07-17T19:15:48Z'} |
| flavor                              | m1.small (1c4ebb14-34b6-4301-bba3-d5bd661945ca)                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| hostId                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| id                                  | 6014d6fc-dff7-493c-96ac-c01dee94a8d6                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| image                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| key_name                            | INE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| name                                | cce-d1test                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| project_id                          | 01b728e67a5a4d88b655e5b3bc232450                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| properties                          |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| status                              | ERROR                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| updated                             | 2019-07-17T19:15:48Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| user_id                             | 70bfd15a9f9a3f32a589a3cfcbcb78886d3be060e961dd38048eef87870202a9                                                                                                                                                                                                                                                                                                                                                                                                                        |
| volumes_attached                    | id='a9fc814e-dfb7-4f0c-b89b-98af3fc38ccb'                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  • The following error is seen on the controllers in /var/log/containers/neutron/server.log:
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers [req-3938763b-d18c-4e47-8073-a4ea6ec8ba80 63abb58b0cfa46bca9f7c2e9772f7183 fdf586078d6f465695db34d793436147 - default default] Mechanism driver 'ovn' failed in update_port_postcommit: RowNotFound: Cannot find Logical_Switch_Port with name=9934adc9-eb68-4acd-9c63-20e9895bb842
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 427, in _call_on_drivers
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 531, in update_port_postcommit
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     self._ovn_client.update_port(port, port_object=original_port)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 440, in update_port
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     ovn_port = self._nb_idl.lookup('Logical_Switch_Port', port['id'])
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 103, in lookup
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     return self._lookup(table, record)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 143, in _lookup
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 63, in row_by_value
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers     raise RowNotFound(table=table, col=column, match=match)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers RowNotFound: Cannot find Logical_Switch_Port with name=9934adc9-eb68-4acd-9c63-20e9895bb842
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers 
  • Almost all nodes have similar error messages in /var/log/messages:
Jul 16 16:13:31 overcloud-control-0 journal: 2019-07-16T16:13:31Z|96095|poll_loop|INFO|wakeup due to [POLLIN] on fd 15 (10.10.10.10:56806<->10.10.10.10:6642) at ../lib/stream-fd.c:157 (100% CPU usage)
  • At some point, performances get so bad that VMs can no longer be spawned and the following error messages will appear in /var/log/containers/nova/nova-compute.log on the compute nodes:
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [req-c3d2fdd0-b514-4514-91d4-622a39d6327d 70bfd15a9f9a3f32a589a3cfcbcb78886d3be060e961dd38048eef87870202a9 01b728e67a5a4d88b655e5b3bc232450 - 0ef23cf14b4a4cecade7d9f1cee1862b 0ef23cf14b4a4cecade7d9f1cee1862b] [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] Instance failed to spawn: VirtualInterfaceCreateException: Virtual Interface creation failed
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] Traceback (most recent call last):
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2273, in _build_resources
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]     yield resources
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2053, in _build_and_run_instance
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]     block_device_info=block_device_info)
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3129, in spawn
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]     destroy_disks_on_failure=True)
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5652, in _create_domain_and_network
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6]     raise exception.VirtualInterfaceCreateException()
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] VirtualInterfaceCreateException: Virtual Interface creation failed

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)
  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In