Horizon UI and CLI not responding

Issue

• The Horizon UI and CLI in our internal implementation of OpenStack are not responding.

• The following is from our keystone logs:

2019-11-21 15:40:36.592 580262 WARNING oslo_db.sqlalchemy.engines [req-484a7999-413f-49bc-aa35-d8ffee962e9d - - - - -] SQL connection failed. -17 attempts left.

• The following INFO logs are returned by galera:

Nov 21 15:37:01 openstack-control-0 galera(galera)[620388]: INFO: Waiting on node <openstack-control-1> to report database status before Master instances can start.
Nov 21 15:45:53 openstack-control-0  galera(galera)[143450]: INFO: Waiting on node <openstack-control-1> to report database status before Master instances can start.

• pcs status returns an output similar to this one:

[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.16-12.el7_4.4-94ff4df) - partition with quorum
Last updated: Fri Nov 22 09:40:56 2019
Last change: Thu Nov 21 15:38:47 2019 by hacluster via crmd on overcloud-controller-2

3 nodes configured
20 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-10.101.224.5        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.101.226.5        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
     Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 ip-10.101.225.6        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-2 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 ip-10.101.225.5        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0
 my-ipmilan-for-controller-0    (stonith:fence_ipmilan):        Started overcloud-controller-2
 my-ipmilan-for-controller-1    (stonith:fence_ipmilan):        Started overcloud-controller-0
 my-ipmilan-for-controller-2    (stonith:fence_ipmilan):        Started overcloud-controller-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled