Compute nodes are timing out when connecting to Neutron services when spawning over 75 instances at one time via nova boot

Solution Unverified - Updated -

Issue

  • When launching a large number of instances simultaneously, numerous neutron timeouts are observed. If enough instances are spawned, some instances will fail to launch.
2016-01-27 21:24:29.497 15442 ERROR nova.network.neutronv2.api [-] [instance: <uuid>] Neutron error creating port on network <uuid>
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] Traceback (most recent call last):
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 215, in _create_port
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     port_id = port_client.create_port(port_req_body)['port']['id']
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     ret = self.function(instance, *args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 322, in create_port
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     return self.post(self.ports_path, body=body)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1325, in post
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     headers=headers, params=params)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     ret = obj(*args, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     content_type=self.content_type())
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 187, in do_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     self.endpoint_url + url, method, **kwargs)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 133, in _cs_request
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>]     raise exceptions.ConnectionFailed(reason=e)
2016-01-27 21:24:29.497 15442 TRACE nova.network.neutronv2.api [instance: <uuid>] ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='<ip_addr>', port=9696): Read timed out. (read timeout=30)

-At the same time, the following error can be seen in server.log:

2016-01-27 21:24:31.095 1146 INFO neutron.wsgi [req-80deb2ec-f8c3-4999-b23f-771a821c0f20 None] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
    _writelines(towrite)
  File "/usr/lib64/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 32] Broken pipe
2016-01-27 21:24:31.096 1146 INFO neutron.wsgi [req-80deb2ec-f8c3-4999-b23f-771a821c0f20 None] <ip_addr> - - [27/Jan/2016 21:24:31] "POST /v2.0/ports.json HTTP/1.1" 201 0 30.163169

Environment

  • Red Hat OpenStack Platform 6.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content