`versioned_notifications.info` queue grows with no consumer
Issue
DISCLAIMER
Be careful with the proposed workaround below!
If you apply the workaround on top of templates that have already been fixed (see the list/BZs below), that will break your deployment, i.e. the openstack overcloud deploy... command will fail, and consequently that will fail your stack.
You can track the progress of this Bug here:
Bug 1558901 - [OSP10][Deployment][undercloud] notification_format in nova.conf should be set to unversioned as there is no consumer for versioned
Bug 1594764 - [OSP12][Deployment][undercloud] notification_format in nova.conf should be set to unversioned as there is no consumer for versioned
Bug 1594763 - [OSP13][Deployment][undercloud] notification_format in nova.conf should be set to unversioned as there is no consumer for versioned
Bug 1579316 - [OSP10] notification_format in nova.conf should be set to unversioned as there is no consumer for versioned
Bug 1478274 - [OSP11] notification_format in nova.conf should be set to unversioned as there is no consumer for versioned
- Fixed in: openstack-tripleo-heat-templates-6.2.7-6.el7ost
Bug 1551137 - [OSP12] Queue versioned_notifications.info not found
- Fixed in: openstack-tripleo-heat-templates-7.0.9-1.el7ost
At the time of writing, we already have this fixed in the latest heat templates for RHOSP-11 (Overcloud nodes) and RHOSP-12 (Overcloud nodes).
WORKAROUND
In a fresh installed OSP environment when instance get spawned we see e.g. the computes send messages to notifications.info and versioned_notifications.info queue, but there is no consumer and the queue grows:
2017-08-03 13:24:31.179 17539 DEBUG oslo_messaging._drivers.amqpdriver [req-228730b4-8c81-4bb8-bbae-1a19803f40b1 64a619d8582c41e4881c44d2c3949cfe 7d90d27a4db541638e370df4a795685c - - -] CAST unique_id: d12f778351f24ca6928cc875207eb321 NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:473
2017-08-03 13:24:31.190 17539 DEBUG oslo_messaging._drivers.amqpdriver [req-228730b4-8c81-4bb8-bbae-1a19803f40b1 64a619d8582c41e4881c44d2c3949cfe 7d90d27a4db541638e370df4a795685c - - -] CAST unique_id: bc352c35adbc494296d0e4122367f32b NOTIFY exchange 'nova' topic 'versioned_notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:473
# rabbitmqctl list_queues name consumers messages| egrep -v "[[:space:]]0$"
Listing queues ...
versioned_notifications.info 0 133
Another symptom for this issue would be when running nova commands such as creating new flavors. The commands would either take up to a minute to complete or would reach a timeout.
Environment
- Red Hat OpenStack Platform 10.0
- Red Hat OpenStack Platform 11.0
- Red Hat OpenStack Platform 12.0
- Red Hat OpenStack Platform 13.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
