Instance creation failures - RabbitMQ related

Issue

• Intermittently new virtual instance creations fail at/before the scheduling step in the provisioning process. The issue seems to be related to messages or message queues being lost or deleted. An example error message from Horizon is

Error: Failed to launch instance "session-service1": Please try again later [Error: Timed out waiting for a reply to message ID ab58af7d4a334e738cbdc412f40b718c].

• An example error message from the /var/log/nova/nova-conductor.log is:

ERROR oslo.messaging._drivers.impl_rabbit [req-...some_request_id_number.... ] Failed to consume message from queue: [Errno 32] Broken pipe

• This error seems to last in the range of a few seconds to minutes, then retrying the same deployment succeeds. This error has occurred across all controllers, so it is not isolated to a single controller.