Rabbitmq instances are randomly closing sockets

Solution In Progress - Updated -

Issue

Issues with rabbitmq disconnects in multiple environments including a large 700+ node environment. In which they are seeing the follow such errors on 20-40 nodes at a time.

2016-05-11 18:36:37.492 266925 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2016-05-11 18:36:37.492 266925 TRACE root Traceback (most recent call last):
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2016-05-11 18:36:37.492 266925 TRACE root     return infunc(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2016-05-11 18:36:37.492 266925 TRACE root     self.consume()
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2016-05-11 18:36:37.492 266925 TRACE root     six.next(it)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2016-05-11 18:36:37.492 266925 TRACE root     yield self.ensure(_error_callback, _consume)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2016-05-11 18:36:37.492 266925 TRACE root     return method(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2016-05-11 18:36:37.492 266925 TRACE root     queues_tail.consume(nowait=False)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2016-05-11 18:36:37.492 266925 TRACE root     self.queue.consume(*args, callback=_callback, **options)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 611, in consume
2016-05-11 18:36:37.492 266925 TRACE root     nowait=nowait)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1787, in basic_consume
2016-05-11 18:36:37.492 266925 TRACE root     (60, 21),  # Channel.basic_consume_ok
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2016-05-11 18:36:37.492 266925 TRACE root     return self.dispatch_method(method_sig, args, content)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-05-11 18:36:37.492 266925 TRACE root     return amqp_method(self, args)
2016-05-11 18:36:37.492 266925 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2016-05-11 18:36:37.492 266925 TRACE root     reply_code, reply_text, (class_id, method_id), ChannelError,
2016-05-11 18:36:37.492 266925 TRACE root NotFound: Basic.consume: (404) NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_b830c7c7b118442bb43c994209cc30a6' in vhost '/'

Environment

*Red Hat Enterprise Linux Openstack Platform 5 on Red Hat Enterprise Linux 7 (OSP5 on RHEL7).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content