Rabbitmq instances are randomly closing sockets

Solution In Progress - Updated -

Issue

Issues with rabbitmq disconnects in multiple environments including a large 700+ node environment. In which they are seeing the follow such errors on 20-40 nodes at a time.

2016-05-11 18:36:37.492 266925 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2016-05-11 18:36:37.492 266925 TRACE root Traceback (most recent call last):
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2016-05-11 18:36:37.492 266925 TRACE root     return infunc(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2016-05-11 18:36:37.492 266925 TRACE root     self.consume()
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2016-05-11 18:36:37.492 266925 TRACE root     six.next(it)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2016-05-11 18:36:37.492 266925 TRACE root     yield self.ensure(_error_callback, _consume)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2016-05-11 18:36:37.492 266925 TRACE root     return method(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2016-05-11 18:36:37.492 266925 TRACE root     queues_tail.consume(nowait=False)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2016-05-11 18:36:37.492 266925 TRACE root     self.queue.consume(*args, callback=_callback, **options)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 611, in consume
2016-05-11 18:36:37.492 266925 TRACE root     nowait=nowait)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1787, in basic_consume
2016-05-11 18:36:37.492 266925 TRACE root     (60, 21),  # Channel.basic_consume_ok
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2016-05-11 18:36:37.492 266925 TRACE root     return self.dispatch_method(method_sig, args, content)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-05-11 18:36:37.492 266925 TRACE root     return amqp_method(self, args)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2016-05-11 18:36:37.492 266925 TRACE root     reply_code, reply_text, (class_id, method_id), ChannelError,
2016-05-11 18:36:37.492 266925 TRACE root NotFound: Basic.consume: (404) NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_b830c7c7b118442bb43c994209cc30a6' in vhost '/'

Environment

*Red Hat Enterprise Linux Openstack Platform 5 on Red Hat Enterprise Linux 7 (OSP5 on RHEL7).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In