Unable to launch instances after a while - Failed to allocate the network(s)
Issue
- After having created a lot of VMs, new VMs can no longer be created an the following error message will be seen in
openstack server show:
[stack@undercloud ~]$ openstack server show 6014d6fc-dff7-493c-96ac-c01dee94a8d6
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000629 |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2019-07-17T19:10:05Z |
| fault | {u'message': u'Build of instance 6014d6fc-dff7-493c-96ac-c01dee94a8d6 aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'details': u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1858, in _do_build_and_run_instance\n filter_properties, request_spec)\n File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2113, in _build_and_run_instance\n reason=msg)\n', u'created': u'2019-07-17T19:15:48Z'} |
| flavor | m1.small (1c4ebb14-34b6-4301-bba3-d5bd661945ca) |
| hostId | |
| id | 6014d6fc-dff7-493c-96ac-c01dee94a8d6 |
| image | |
| key_name | INE |
| name | cce-d1test |
| project_id | 01b728e67a5a4d88b655e5b3bc232450 |
| properties | |
| status | ERROR |
| updated | 2019-07-17T19:15:48Z |
| user_id | 70bfd15a9f9a3f32a589a3cfcbcb78886d3be060e961dd38048eef87870202a9 |
| volumes_attached | id='a9fc814e-dfb7-4f0c-b89b-98af3fc38ccb' |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- The following error is seen on the controllers in
/var/log/containers/neutron/server.log:
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers [req-3938763b-d18c-4e47-8073-a4ea6ec8ba80 63abb58b0cfa46bca9f7c2e9772f7183 fdf586078d6f465695db34d793436147 - default default] Mechanism driver 'ovn' failed in update_port_postcommit: RowNotFound: Cannot find Logical_Switch_Port with name=9934adc9-eb68-4acd-9c63-20e9895bb842
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 427, in _call_on_drivers
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py", line 531, in update_port_postcommit
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers self._ovn_client.update_port(port, port_object=original_port)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/networking_ovn/common/ovn_client.py", line 440, in update_port
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers ovn_port = self._nb_idl.lookup('Logical_Switch_Port', port['id'])
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 103, in lookup
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers return self._lookup(table, record)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 143, in _lookup
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers row = idlutils.row_by_value(self, rl.table, rl.column, record)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 63, in row_by_value
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers raise RowNotFound(table=table, col=column, match=match)
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers RowNotFound: Cannot find Logical_Switch_Port with name=9934adc9-eb68-4acd-9c63-20e9895bb842
2019-07-18 15:44:50.671 36 ERROR neutron.plugins.ml2.managers
- Almost all nodes have similar error messages in
/var/log/messages:
Jul 16 16:13:31 overcloud-control-0 journal: 2019-07-16T16:13:31Z|96095|poll_loop|INFO|wakeup due to [POLLIN] on fd 15 (10.10.10.10:56806<->10.10.10.10:6642) at ../lib/stream-fd.c:157 (100% CPU usage)
- At some point, performances get so bad that VMs can no longer be spawned and the following error messages will appear in
/var/log/containers/nova/nova-compute.logon the compute nodes:
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [req-c3d2fdd0-b514-4514-91d4-622a39d6327d 70bfd15a9f9a3f32a589a3cfcbcb78886d3be060e961dd38048eef87870202a9 01b728e67a5a4d88b655e5b3bc232450 - 0ef23cf14b4a4cecade7d9f1cee1862b 0ef23cf14b4a4cecade7d9f1cee1862b] [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] Instance failed to spawn: VirtualInterfaceCreateException: Virtual Interface creation failed
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] Traceback (most recent call last):
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2273, in _build_resources
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] yield resources
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2053, in _build_and_run_instance
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] block_device_info=block_device_info)
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3129, in spawn
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] destroy_disks_on_failure=True)
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5652, in _create_domain_and_network
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] raise exception.VirtualInterfaceCreateException()
2019-07-17 19:15:42.696 1 ERROR nova.compute.manager [instance: 6014d6fc-dff7-493c-96ac-c01dee94a8d6] VirtualInterfaceCreateException: Virtual Interface creation failed
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
- Red Hat OpenStack Platform 16.1 (RHOSP)
- Red Hat OpenStack Platform 16.2 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
