Disabling telemetry does not seem to work on Red Hat OpenStack Platform

Issue

After implementing suggested changes from https://access.redhat.com/solutions/3024461 and https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/deployment_recommendations_for_specific_red_hat_openstack_platform_services/#svc-telemetry-small one can still see notifications.* queues growing:

[root@nspregr-controller-0 ~]# rabbitmqctl list_queues name consumers messages | grep notifications
notifications.info      0       225450
notifications.error     0       438
[root@nspregr-controller-0 ~]# sleep 120; rabbitmqctl list_queues name consumers messages | grep notifications
notifications.info      0       225645
notifications.error     0       438

Looking at the messages, one can see that they are of the following event types:

orchestration.stack.*
floatingip.{create,delete}.{start,end}
port.{create,delete}.{start,end}
{create,delete}.{start,end}

While driver = noop is set in /etc/neutron/neutron.conf properly from the director deploy, it seems that the /usr/share/neutron/neutron-dist.conf with '[DEFAULT]/notification_driver = messaging' is combined with the 'noop' value. this can be verified with the following python script:

cat <<EOF>neutron_conf_verification.sh
#!/bin/env python
args='/usr/bin/python2 /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log'.split(' ')
from oslo_config import cfg
from neutron.common import config
from neutron.common import rpc as n_rpc
config.init(args[2:])

for k, v in cfg.CONF.oslo_messaging_notifications.iteritems(): print k, v
EOF
chmod +x neutron_conf_verification.sh
./neutron_conf_verification.sh

The output of which is:

transport_url None
topics ['notifications']
driver ['messaging', 'noop']

The above settings being improperly configured leads to undesired messages dispatched to the notifications.info queues, which grow unbounded.

What information can you provide around timeframes and the business impact?

Rabbitmq may enter into a bad state and this can cause neutron to have issues. The notifications.info queue may disappear and will not return until restarting rabbitmq, causing timeouts when to create new VMs/ports/floating IP associations.

From /var/log/neutron/server.log:

2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     queue.declare()
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 522, in declare
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     self.queue_declare(nowait, passive=False)
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 548, in queue_declare
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     nowait=nowait)
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1258, in queue_declare
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     (50, 11),  # Channel.queue_declare_ok
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     return self.dispatch_method(method_sig, args, content)
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     return amqp_method(self, args)
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging     reply_code, reply_text, (class_id, method_id), ChannelError,
2019-01-01 17:27:49.454 13013 ERROR oslo_messaging.notify.messaging NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/' due to timeout