Rabbitmq instances are randomly closing sockets
Issue
Issues with rabbitmq disconnects in multiple environments including a large 700+ node environment. In which they are seeing the follow such errors on 20-40 nodes at a time.
2016-05-11 18:36:37.492 266925 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2016-05-11 18:36:37.492 266925 TRACE root Traceback (most recent call last):
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2016-05-11 18:36:37.492 266925 TRACE root return infunc(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2016-05-11 18:36:37.492 266925 TRACE root self.consume()
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2016-05-11 18:36:37.492 266925 TRACE root six.next(it)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2016-05-11 18:36:37.492 266925 TRACE root yield self.ensure(_error_callback, _consume)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2016-05-11 18:36:37.492 266925 TRACE root return method(*args, **kwargs)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2016-05-11 18:36:37.492 266925 TRACE root queues_tail.consume(nowait=False)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2016-05-11 18:36:37.492 266925 TRACE root self.queue.consume(*args, callback=_callback, **options)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 611, in consume
2016-05-11 18:36:37.492 266925 TRACE root nowait=nowait)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1787, in basic_consume
2016-05-11 18:36:37.492 266925 TRACE root (60, 21), # Channel.basic_consume_ok
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2016-05-11 18:36:37.492 266925 TRACE root return self.dispatch_method(method_sig, args, content)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-05-11 18:36:37.492 266925 TRACE root return amqp_method(self, args)
2016-05-11 18:36:37.492 266925 TRACE root File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2016-05-11 18:36:37.492 266925 TRACE root reply_code, reply_text, (class_id, method_id), ChannelError,
2016-05-11 18:36:37.492 266925 TRACE root NotFound: Basic.consume: (404) NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_b830c7c7b118442bb43c994209cc30a6' in vhost '/'
Environment
*Red Hat Enterprise Linux Openstack Platform 5 on Red Hat Enterprise Linux 7 (OSP5 on RHEL7).
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.