Intermittent AMQP errors in neutron logs
Issue
- Seeing below errors in neutron logs intermittently which is causing network instability:
ERROR oslo.messaging._drivers.impl_rabbit [-] [201eb7fb-5cee-432f-ac8e-0aa3216defe2] AMQP server on 192.x.x.9:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None
ERROR oslo.messaging._drivers.impl_rabbit [-] [201eb7fb-5cee-432f-ac8e-0aa3216defe2] AMQP server on 192.x.x.9:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None
ERROR neutron.common.rpc [-] Timeout in RPC method get_ports. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
ERROR neutron.common.rpc [-] Timeout in RPC method get_ports. Waiting for 42 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
ERROR neutron.agent.metadata.agent [-] Unexpected error.
ERROR neutron.agent.metadata.agent Traceback (most recent call last):
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 88, in __call__
ERROR neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 161, in _get_instance_and_tenant_id
ERROR neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 149, in _get_ports
ERROR neutron.agent.metadata.agent networks = self._get_router_networks(router_id)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 154, in __call__
ERROR neutron.agent.metadata.agent return self._get_from_cache(target_self, *args, **kwargs)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 130, in _get_from_cache
ERROR neutron.agent.metadata.agent item = self.func(target_self, *args, **kwargs)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 123, in _get_router_networks
ERROR neutron.agent.metadata.agent internal_ports = self._get_ports_from_server(router_id=router_id)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 105, in _get_ports_from_server
ERROR neutron.agent.metadata.agent return self.plugin_rpc.get_ports(self.context, filters)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 71, in get_ports
ERROR neutron.agent.metadata.agent return cctxt.call(context, 'get_ports', filters=filters)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 185, in call
ERROR neutron.agent.metadata.agent time.sleep(wait)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR neutron.agent.metadata.agent self.force_reraise()
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR neutron.agent.metadata.agent six.reraise(self.type_, self.value, self.tb)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call
ERROR neutron.agent.metadata.agent return self._original_context.call(ctxt, method, **kwargs)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
ERROR neutron.agent.metadata.agent retry=self.retry)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
ERROR neutron.agent.metadata.agent timeout=timeout, retry=retry)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
ERROR neutron.agent.metadata.agent retry=retry)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
ERROR neutron.agent.metadata.agent result = self._waiter.wait(msg_id, timeout)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
ERROR neutron.agent.metadata.agent message = self.waiters.get(msg_id, timeout=timeout)
ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
ERROR neutron.agent.metadata.agent 'to message ID %s' % msg_id)
ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID f705f151a6a4439aac2e928b03ae5ef9
ERROR neutron.agent.metadata.agent
Environment
- Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
