Check the current status of the overcloud stack on the Undercloud:

$ source stackrc
$ heat stack-list --show-nested

The overcloud stack and its subsequent child stacks should have either a CREATE_COMPLETE or UPDATE_COMPLETE.

Perform a backup of the Undercloud databases:

$ mkdir /home/stack/backup
$ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz
$ sudo systemctl stop openstack-ironic-api.service openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service
$ sudo cp /var/lib/ironic-inspector/inspector.sqlite /home/stack/backup
$ sudo systemctl start openstack-ironic-api.service openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service

Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the Pacemaker status:

$ ssh heat-admin@192.168.0.47 'sudo pcs status'

The output should show all services running on the existing nodes and stopped on the failed node.

Check the following parameters on each node of the Overcloud’s MariaDB cluster:

Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the status

$ ssh heat-admin@192.168.0.47 "sudo rabbitmqctl cluster_status"

The running_nodes key should only show the two available nodes and not the failed node.

Disable fencing, if enabled. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to disable fencing:

$ ssh heat-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"

Check the fencing status with the following command:

$ ssh heat-admin@192.168.0.47 "sudo pcs property show stonith-enabled"

Check the nova-compute service on the director node:

$ sudo systemctl status openstack-nova-compute
$ nova hypervisor-list

The output should show all non-maintenance mode nodes as up.

Make sure all Undercloud services are running:

$ sudo systemctl -t service

Get a list of IP addresses for the Controller nodes. For example:

[stack@director ~]$ nova list
... +------------------------+ ... +-------------------------+
... | Name                   | ... | Networks                |
... +------------------------+ ... +-------------------------+
... | overcloud-compute-0    | ... | ctlplane=192.168.0.44   |
... | overcloud-controller-0 | ... | ctlplane=192.168.0.47   |
... | overcloud-controller-2 | ... | ctlplane=192.168.0.46   |
... | overcloud-controller-3 | ... | ctlplane=192.168.0.48   |
... +------------------------+ ... +-------------------------+

Check the nodeid value of the removed node in the /etc/corosync/corosync.conf file on an existing node. For example, the existing node is overcloud-controller-0 at 192.168.0.47:

[stack@director ~]$ ssh heat-admin@192.168.0.47 "sudo cat /etc/corosync/corosync.conf"

This displays a nodelist that contains the ID for the removed node (overcloud-controller-1):

nodelist {
  node {
    ring0_addr: overcloud-controller-0
    nodeid: 1
  }
  node {
    ring0_addr: overcloud-controller-1
    nodeid: 2
  }
  node {
    ring0_addr: overcloud-controller-2
    nodeid: 3
  }
}

Note the nodeid value of the removed node for later. In this example, it is 2.

Delete the failed node from the Corosync configuration on each node and restart Corosync. For this example, log into overcloud-controller-0 and overcloud-controller-2 and run the following commands:

[stack@director] ssh heat-admin@192.168.0.47 "sudo pcs cluster localnode remove overcloud-controller-1"
[stack@director] ssh heat-admin@192.168.0.47 "sudo pcs cluster reload corosync"
[stack@director] ssh heat-admin@192.168.0.46 "sudo pcs cluster localnode remove overcloud-controller-1"
[stack@director] ssh heat-admin@192.168.0.46 "sudo pcs cluster reload corosync"

Log into one of the remaining nodes and delete the node from the cluster with the crm_node command:

[stack@director] ssh heat-admin@192.168.0.47
[heat-admin@overcloud-controller-0 ~]$ sudo crm_node -R overcloud-controller-1 --force

Stay logged into this node.

Delete the failed node from the RabbitMQ cluster:

[heat-admin@overcloud-controller-0 ~]$ sudo rabbitmqctl forget_cluster_node rabbit@overcloud-controller-1

Delete the failed node from MongoDB. First, find the IP address for the node’s Interal API connection.

[heat-admin@overcloud-controller-0 ~]$ sudo netstat -tulnp | grep 27017
tcp        0      0 192.168.0.47:27017    0.0.0.0:*               LISTEN      13415/mongod

Check that the node is the primary replica set:

[root@overcloud-controller-0 ~]# echo "db.isMaster()" | mongo --host 192.168.0.47:27017
MongoDB shell version: 2.6.11
connecting to: 192.168.0.47:27017/echo
{
  "setName" : "tripleo",
  "setVersion" : 1,
  "ismaster" : true,
  "secondary" : false,
  "hosts" : [
    "192.168.0.47:27017",
    "192.168.0.46:27017",
    "192.168.0.45:27017"
  ],
  "primary" : "192.168.0.47:27017",
  "me" : "192.168.0.47:27017",
  "electionId" : ObjectId("575919933ea8637676159d28"),
  "maxBsonObjectSize" : 16777216,
  "maxMessageSizeBytes" : 48000000,
  "maxWriteBatchSize" : 1000,
  "localTime" : ISODate("2016-06-09T09:02:43.340Z"),
  "maxWireVersion" : 2,
  "minWireVersion" : 0,
  "ok" : 1
}
bye

This should indicate if the current node is the primary. If not, use the IP address of the node indicated in the primary key.

Connect to MongoDB on the primary node:

[heat-admin@overcloud-controller-0 ~]$ mongo --host 192.168.0.47
MongoDB shell version: 2.6.9
connecting to: 192.168.0.47:27017/test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
tripleo:PRIMARY>

Check the status of the MongoDB cluster:

tripleo:PRIMARY> rs.status()

Identify the node using the _id key and remove the failed node using the name key. In this case, we remove Node 1, which has 192.168.0.45:27017 for name:

tripleo:PRIMARY> rs.remove('192.168.0.45:27017')

"replSetReconfig command must be sent to the current replica set primary."

2016-05-07T03:57:19.541+0000 DBClientCursor::init call() failed
2016-05-07T03:57:19.543+0000 Error: error doing query: failed at src/mongo/shell/query.js:81
2016-05-07T03:57:19.545+0000 trying reconnect to 192.168.0.47:27017 (192.168.0.47) failed
2016-05-07T03:57:19.547+0000 reconnect 192.168.0.47:27017 (192.168.0.47) ok

Exit MongoDB:

tripleo:PRIMARY> exit

Update list of nodes in the Galera cluster:

[heat-admin@overcloud-controller-0 ~]$ sudo pcs resource update galera wsrep_cluster_address=gcomm://overcloud-controller-0,overcloud-controller-3,overcloud-controller-2

Add the new node to the cluster:

[heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster node add overcloud-controller-3

Check the /etc/corosync/corosync.conf file on each node. If the nodeid of the new node is the same as the removed node, update the value to a new nodeid value. For example, the /etc/corosync/corosync.conf file contains an entry for the new node (overcloud-controller-3):

nodelist {
  node {
    ring0_addr: overcloud-controller-0
    nodeid: 1
  }
  node {
    ring0_addr: overcloud-controller-2
    nodeid: 3
  }
  node {
    ring0_addr: overcloud-controller-3
    nodeid: 2
  }
}

Note that in this example, the new node uses the same nodeid of the removed node. Update this value to a unused node ID value. For example:

node {
  ring0_addr: overcloud-controller-3
  nodeid: 4
}

Update this nodeid value on each Controller node’s /etc/corosync/corosync.conf file, including the new node.

Restart the Corosync service on the existing nodes only. For example, on overcloud-controller-0:

[heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster reload corosync

And on overcloud-controller-2:

[heat-admin@overcloud-controller-2 ~]$ sudo pcs cluster reload corosync

Do not run this command on the new node.

Start the new Controller node:

[heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster start overcloud-controller-3

Enable the keystone service on the new node. Copy the /etc/keystone directory from a remaining node to the director host:

[heat-admin@overcloud-controller-0 ~]$ sudo -i
[root@overcloud-controller-0 ~]$ scp -r /etc/keystone stack@192.168.0.1:~/.

Log in to the new Controller node. Remove the /etc/keystone directory from the new Controller node and copy the keystone files from the director host:

[heat-admin@overcloud-controller-3 ~]$ sudo -i
[root@overcloud-controller-3 ~]$ rm -rf /etc/keystone
[root@overcloud-controller-3 ~]$ scp -r stack@192.168.0.1:~/keystone /etc/.
[root@overcloud-controller-3 ~]$ chown -R keystone: /etc/keystone
[root@overcloud-controller-3 ~]$ chown root /etc/keystone/logging.conf /etc/keystone/default_catalog.templates

Edit /etc/keystone/keystone.conf and set the admin_bind_host and public_bind_host parameters to new Controller node’s IP addresses. To find these IP addresses, use the ip addr command and look for the IP address within the following networks:

For example, if the Provisioning network uses the 192.168.0.0/24 subnet and the Internal API uses the 172.17.0.0/24 subnet, use the following commands to find the node’s IP addresses on those networks:

[root@overcloud-controller-3 ~]$ ip addr | grep "192\.168\.0\..*/24"
[root@overcloud-controller-3 ~]$ ip addr | grep "172\.17\.0\..*/24"

Enable and restart some services through Pacemaker. The cluster is currently in maintenance mode and you will need to temporarily disable it to enable the service. For example:

[heat-admin@overcloud-controller-3 ~]$ sudo pcs property set maintenance-mode=false --wait

Wait until the Galera service starts on all nodes.

[heat-admin@overcloud-controller-3 ~]$ sudo pcs status | grep galera -A1
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-2 overcloud-controller-3 ]

If need be, perform a cleanup on the new node:

[heat-admin@overcloud-controller-3 ~]$ sudo pcs resource cleanup galera --node overcloud-controller-3

Wait until the httpd service starts on all nodes.

[heat-admin@overcloud-controller-3 ~]$ sudo pcs status | grep httpd -A1
Clone Set: httpd-clone [httpd]
Started: [ overcloud-controller-0 overcloud-controller-2 overcloud-controller-3 ]

If need be, perform a cleanup on the new node:

[heat-admin@overcloud-controller-3 ~]$ sudo pcs resource cleanup httpd --node overcloud-controller-3

Switch the cluster back into maintenance mode:

[heat-admin@overcloud-controller-3 ~]$ sudo pcs property set maintenance-mode=true --wait