Unable to delete or show stack information

Issue

• End user is reporting stack deploy failed. We can see in openstack stack list the stack that failed: stack-vnf (96b65910-8935-4e22-b7d3-2062b4cd2698). However, when we use the command openstack stack show 96b65910-8935-4e22-b7d3-2062b4cd2698, the request times out with the error message: ERROR: Timed out waiting for a reply to message ID cd3cb5cfff94454bacb474181da1dcb3

• We can confirm that other OSP commands work fine. (user list, flavor list, server list, subnet list, etc.) We did see one occurrance of openstack stack list producing the error:

[stack@undercloud-0-rhosp10 ~]$ openstack stack list
ERROR: Timed out waiting for a reply to message ID c484741172bd48998e220b96a39b4ee9

• The following errors are seen in /var/log/heat/heat-api.log:

2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi [req-981d59c3-7aba-4b9c-9ea4-da075f5ba040 bd0b103077624f24b966e623716d321d 87e55c90ba20400e866480f5a989d52a - default default] Unexpected error occurred serving API: Timed out waiting for a reply to message ID f3456c0dd42c4c0c823aade12ee0465d
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi Traceback (most recent call last):
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/common/wsgi.py", line 842, in __call__
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     request, **action_args)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/common/wsgi.py", line 916, in dispatch
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     return method(*args, **kwargs)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/api/openstack/v1/util.py", line 38, in handle_stack_method
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     return handler(controller, req, **kwargs)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/api/openstack/v1/stacks.py", line 416, in lookup
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     stack_name)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 119, in identify_stack
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     stack_name=stack_name))
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 84, in call
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     return client.call(ctxt, method, **kwargs)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 428, in call
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     return self.prepare().call(ctxt, method, **kwargs)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     retry=self.retry)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     timeout=timeout, retry=retry)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     retry=retry)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     result = self._waiter.wait(msg_id, timeout)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi     'to message ID %s' % msg_id)
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi MessagingTimeout: Timed out waiting for a reply to message ID f3456c0dd42c4c0c823aade12ee0465d
2019-10-04 13:15:08.395 4783 ERROR heat.common.wsgi