Openstack services are slowing down while rabbitmq messages are growing up without being consumed.

Solution Verified - Updated -

Issue

  • In my openstack deployment, I can see that messages in certain rabbitmq queues are growing without being consumed. Some of these queues have no consumers as well. The more the messages in the queue, the more it starts slowing down operations from openstack services. Eg: Below shows below queues have more than 1000 messages.
# rabbitmqctl list_queues |  awk 'int($2)>=1000'
cinder-scheduler_fanout_9aed9fbc3d4249289f2cb5ea04c062ab        8163
cinder-scheduler_fanout_b7a2e488f3ed4e1587b959f9ac255b93        8159
cinder-scheduler_fanout_ea9c69fb630f41b2ae6120eba3cd43e0        8163
cinder-scheduler_fanout_fa01ec50173647a7b5c031230956eea9        8148

But there are no consumers for those messages.

# rabbitmqctl list_queues name messages consumers | grep cinder-scheduler
cinder-scheduler        0       3
cinder-scheduler.ha-controller  0       3
cinder-scheduler_fanout_9aed9fbc3d4249289f2cb5ea04c062ab        8145    0
cinder-scheduler_fanout_b7a2e488f3ed4e1587b959f9ac255b93        8141    0
cinder-scheduler_fanout_ea9c69fb630f41b2ae6120eba3cd43e0        8145    0
cinder-scheduler_fanout_fa01ec50173647a7b5c031230956eea9        8130    0

Why does this happen? What should I do when I come across such a problem on my openstack deployment which slows down all openstack operations that uses message queue?

Environment

  • Red Hat Enterprise Linux Openstack Platfrom 7.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content