RBD flattening in Cinder is causing RabbitMQ resets to cinder-volume

Issue

• When creating a cinder volume with the following command, we are noticing rabbit is getting reset from the cinder-volume service. This is leading to the volume going into an ERROR status state:

openstack volume create --source d5b6f370-11b9-4561-b8e6-ec40bf049cca --type volumes test

• The following logs are seen in the cinder-volume.log file:

2020-10-06 15:15:50.746 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:50.846 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] creating snapshot='volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:588
2020-10-06 15:15:51.585 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] cloning 'volume-d5b6f370-11b9-4561-b8e6-ec40bf049cca@volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' to 'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:597
2020-10-06 15:15:52.003 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Volume volume-eb0c640e-f22e-4b49-8675-cff8bc39ea26.deleted is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:972
2020-10-06 15:15:52.005 55 INFO cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] maximum clone depth (5) has been reached - flattening dest volume
2020-10-06 15:15:52.091 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] flattening dest volume volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72 create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:625
2020-10-06 15:15:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.619 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.656 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:41.979 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.017 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.052 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.661 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:41.976 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.012 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.047 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.666 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:41.981 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.016 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.049 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.624 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.657 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:19:05.575 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] remove temporary snap volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:640
2020-10-06 15:19:05.688 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] clone created successfully create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:666
2020-10-06 15:19:05.690 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5b07b48e-e622-4603-b45b-81ac5099ea67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72', 'source_volid': u'd5b6f370-11b9-4561-b8e6-ec40bf049cca', 'source_volstatus': u'in-use'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183
2020-10-06 15:19:05.691 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (3addaf14-ccd7-4889-8dc0-1b2b4d880911) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194
2020-10-06 15:19:05.694 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.696 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in _consume
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     raise self.connection.recoverable_connection_errors[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] AMQP server on overcloud-controller-1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer

• The following logs are seen in the rabbitmq log files:

=ERROR REPORT==== 6-Oct-2020::16:55:41 ===
closing AMQP connection <0.20256.0> (10.10.10.10:39952 -> 10.10.10.11:5672 - cinder-volume:55:b8e9ab46-ab18-45e6-835d-78aa1756bd2e):
missed heartbeats from client, timeout: 60s