Intermittent AMQP errors in neutron logs

Issue

• Seeing below errors in neutron logs intermittently which is causing network instability:

ERROR oslo.messaging._drivers.impl_rabbit [-] [201eb7fb-5cee-432f-ac8e-0aa3216defe2] AMQP server on 192.x.x.9:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None
ERROR oslo.messaging._drivers.impl_rabbit [-] [201eb7fb-5cee-432f-ac8e-0aa3216defe2] AMQP server on 192.x.x.9:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None
ERROR neutron.common.rpc [-] Timeout in RPC method get_ports. Waiting for 40 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
ERROR neutron.common.rpc [-] Timeout in RPC method get_ports. Waiting for 42 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.
ERROR neutron.agent.metadata.agent [-] Unexpected error.
ERROR neutron.agent.metadata.agent Traceback (most recent call last):
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 88, in __call__
ERROR neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 161, in _get_instance_and_tenant_id
ERROR neutron.agent.metadata.agent     ports = self._get_ports(remote_address, network_id, router_id)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 149, in _get_ports
ERROR neutron.agent.metadata.agent     networks = self._get_router_networks(router_id)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 154, in __call__
ERROR neutron.agent.metadata.agent     return self._get_from_cache(target_self, *args, **kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/cache_utils.py", line 130, in _get_from_cache
ERROR neutron.agent.metadata.agent     item = self.func(target_self, *args, **kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 123, in _get_router_networks
ERROR neutron.agent.metadata.agent     internal_ports = self._get_ports_from_server(router_id=router_id)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 105, in _get_ports_from_server
ERROR neutron.agent.metadata.agent     return self.plugin_rpc.get_ports(self.context, filters)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 71, in get_ports
ERROR neutron.agent.metadata.agent     return cctxt.call(context, 'get_ports', filters=filters)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 185, in call
ERROR neutron.agent.metadata.agent     time.sleep(wait)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR neutron.agent.metadata.agent     self.force_reraise()
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR neutron.agent.metadata.agent     six.reraise(self.type_, self.value, self.tb)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call
ERROR neutron.agent.metadata.agent     return self._original_context.call(ctxt, method, **kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
ERROR neutron.agent.metadata.agent     retry=self.retry)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
ERROR neutron.agent.metadata.agent     timeout=timeout, retry=retry)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
ERROR neutron.agent.metadata.agent     retry=retry)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
ERROR neutron.agent.metadata.agent     result = self._waiter.wait(msg_id, timeout)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
ERROR neutron.agent.metadata.agent     message = self.waiters.get(msg_id, timeout=timeout)
ERROR neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
ERROR neutron.agent.metadata.agent     'to message ID %s' % msg_id)
ERROR neutron.agent.metadata.agent MessagingTimeout: Timed out waiting for a reply to message ID f705f151a6a4439aac2e928b03ae5ef9
ERROR neutron.agent.metadata.agent