Compute nodes are timing out when connecting to Neutron services when spawning over 75 instances at one time via nova boot
Issue
- When launching a large number of instances simultaneously, numerous neutron timeouts are observed. If enough instances are spawned, some instances will fail to launch.
2016-01-27 21:24:29.497 15442 ERROR nova.network.neutronv2.api [-] [instance: <uuid>] Neutron error creating port on network <uuid>
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] Traceback (most recent call last):
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 215, in _create_port
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] port_id = port_client.create_port(port_req_body)['port']['id']
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ret = self.function(instance, *args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 322, in create_port
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] return self.post(self.ports_path, body=body)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1325, in post
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] headers=headers, params=params)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] content_type=self.content_type())
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 187, in do_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] self.endpoint_url + url, method, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 133, in _cs_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] raise exceptions.ConnectionFailed(reason=e)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='<ip_addr>', port=9696): Read timed out. (read timeout=30)
-At the same time, the following error can be seen in server.log:
2016-01-27 21:24:31.095 1146 INFO neutron.wsgi [req-80deb2ec-f8c3-4999-b23f-771a821c0f20 None] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response
write(b''.join(towrite))
File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
_writelines(towrite)
File "/usr/lib64/python2.7/socket.py", line 334, in writelines
self.flush()
File "/usr/lib64/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
tail = self.send(data, flags)
File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe
2016-01-27 21:24:31.096 1146 INFO neutron.wsgi [req-80deb2ec-f8c3-4999-b23f-771a821c0f20 None] <ip_addr> - - [27/Jan/2016 21:24:31] "POST /v2.0/ports.json HTTP/1.1" 201 0 30.163169
Environment
- Red Hat OpenStack Platform 6.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.