Neutron Timeout Errors when spawning VMs.

Solution In Progress - Updated -

Issue

  • Instances being spawned into error state due to:
Request to http://10.10.10.10:9696/v2.0/ports?device_id=48713308-741f-4895-9ab5-0aaf93de6e46 timed out
  • Here is an example of a full trace:
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [req-668b17e5-995b-4239-a59e-1ebc084e43ee - - - - -] [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee] An error occurred while refreshing the network cache.: ConnectTimeout: Request to http://10.10.10.10:9696/v2.0/subnets?id=47fc78aa-3e16-407d-bd78-e16ad2df4a77 timed out
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee] Traceback (most recent call last):
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6799, in _heal_instance_info_cache
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     context, instance, force_refresh=True)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 249, in get_instance_nw_info
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     result = self._get_instance_nw_info(context, instance, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1463, in _get_instance_nw_info
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     force_refresh=force_refresh)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2676, in _build_network_info_model
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     preexisting_port_ids)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2524, in _build_vif_model
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     network_IPs, client)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2408, in _nw_info_get_subnets
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     subnets = self._get_subnets_from_port(context, port, client)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 2737, in _get_subnets_from_port
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     data = client.list_subnets(**search_opts)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 116, in wrapper
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     ret = obj(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 831, in list_subnets
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     **_params)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 116, in wrapper
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     ret = obj(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     for r in self._pagination(collection, path, **params):
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     res = self.get(path, params=params)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 116, in wrapper
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     ret = obj(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     headers=headers, params=params)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 116, in wrapper
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     ret = obj(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     headers=headers, params=params)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 116, in wrapper
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     ret = obj(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     headers=headers)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 343, in do_request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     return self.request(url, method, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 331, in request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     resp = super(SessionClient, self).request(*args, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     return self.session.request(url, method, **kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 698, in request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     resp = send(**kwargs)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 763, in _send_request
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee]     raise exceptions.ConnectTimeout(msg)
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee] ConnectTimeout: Request to http://10.10.10.10:9696/v2.0/subnets?id=47fc78aa-3e16-407d-bd78-e16ad2df4a77 timed out
2020-09-09 05:54:58.331 1 ERROR nova.compute.manager [instance: e85cc05b-b214-4c1b-ad82-8eb9c4b3e5ee] 
  • In /var/log/containers/neutron/server.log we see the request completed without error:
2020-09-09 05:55:05.472 58 INFO neutron.wsgi [req-23b8f857-32cc-4551-8eff-0f1ab5f90bcc 76aba00e43ce42849fa863ff634a5164 b7a85cca2a404c1aaf8087f985c3feef - default default] 10.150.2.26 "GET /v2.0/subnets?id=47fc78aa-3e16-407d-bd78-e16ad2df4a77 HTTP/1.1" status: 200  len: 843 time: 37.1405191

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In