Rabbitmq instances are randomly closing sockets
Issue
Issues with rabbitmq disconnects in multiple environments including a large 700+ node environment. In which they are seeing the follow such errors on 20-40 nodes at a time.
2016-05-11 18:36:37.492 266925 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2016-05-11 18:36:37.492 266925 TRACE root Traceback (most recent call last):
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2016-05-11 18:36:37.492 266925 TRACE root return infunc(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2016-05-11 18:36:37.492 266925 TRACE root self.consume()
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2016-05-11 18:36:37.492 266925 TRACE root six.next(it)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2016-05-11 18:36:37.492 266925 TRACE root yield self.ensure(_error_callback, _consume)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2016-05-11 18:36:37.492 266925 TRACE root return method(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2016-05-11 18:36:37.492 266925 TRACE root queues_tail.consume(nowait=False)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2016-05-11 18:36:37.492 266925 TRACE root self.queue.consume(*args, callback=_callback, **options)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 611, in consume
2016-05-11 18:36:37.492 266925 TRACE root nowait=nowait)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1787, in basic_consume
2016-05-11 18:36:37.492 266925 TRACE root (60, 21), # Channel.basic_consume_ok
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2016-05-11 18:36:37.492 266925 TRACE root return self.dispatch_method(method_sig, args, content)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-05-11 18:36:37.492 266925 TRACE root return amqp_method(self, args)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2016-05-11 18:36:37.492 266925 TRACE root reply_code, reply_text, (class_id, method_id), ChannelError,
2016-05-11 18:36:37.492 266925 TRACE root NotFound: Basic.consume: (404) NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_b830c7c7b118442bb43c994209cc30a6' in vhost '/'
Environment
*Red Hat Enterprise Linux Openstack Platform 5 on Red Hat Enterprise Linux 7 (OSP5 on RHEL7).
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
