Create an authentication key on the director node to use with the pacemaker-remote service.

stack@director # dd if=/dev/urandom of=~/authkey bs=4096 count=1

Copy the authentication key to the Compute and Controller nodes:

stack@director # scp authkey heat-admin@node-n:~/
stack@director # ssh heat-admin@node-n:~/
heat-admin@node-n $ sudo mkdir -p --mode=0750 /etc/pacemaker
heat-admin@node-n $ sudo chgrp haclient /etc/pacemaker
heat-admin@node-n $ sudo mv authkey /etc/pacemaker/
heat-admin@node-n $ sudo chown root:haclient /etc/pacemaker/authkey

On each Compute node, enable the pacemaker-remote service and configure the firewall.

heat-admin@compute-n $ sudo systemctl enable pacemaker_remote
heat-admin@compute-n $ sudo systemctl start pacemaker_remote
heat-admin@compute-n $ sudo iptables -I INPUT 11 -p tcp --dport 3121 -j ACCEPT ; /sbin/service iptables save

Confirm that the required versions of the pacemaker (1.1.18-11.el7.x86_64) and resource-agents (3.9.5-124.el7.x86_64) packages are installed on the Controller and Compute nodes.

heat-admin@controller-n $ sudo rpm -qa | egrep '(pacemaker|resource-agents)'

Create a NovaEvacuate active/passive resource with the overcloudrc.v3 file, which provides the auth_url, username, tenant and password values.

stack@director # scp overcloudrc.v3 heat-admin@controller-1:~/
heat-admin@controller-1 $ . ~/overcloudrc.v3
heat-admin@controller-1 $ sudo pcs resource create nova-evacuate ocf:openstack:NovaEvacuate auth_url=$OS_AUTH_URL username=$OS_USERNAME password=$OS_PASSWORD tenant_name=$OS_TENANT_NAME project_domain=$OS_PROJECT_DOMAIN_NAME user_domain=$OS_USER_DOMAIN_NAME

Make sure that nova-evacuate runs only on non-Compute nodes.

heat-admin@controller-1 $ pcs constraint location nova-evacuate rule resource-discovery=never score=-INFINITY osprole eq compute

Confirm that nova-evacuate is started after the floating IP resources, OpenStack Image Service (glance), OpenStack Networking (neutron), and Compute (nova) services.

heat-admin@controller-1 $ for i in $(sudo pcs status | grep IP | awk '{ print $1 }\'); do sudo pcs constraint order start $i then nova-evacuate ; done

Create a list of the current controllers from the cibadmin data.

heat-admin@controller-1 $ controllers=$(sudo cibadmin -Q -o nodes | grep uname | sed s/.\*uname..// | awk -F\" '{print $1}')
heat-admin@controller-1 $ echo $controllers

Use the list you created in the previous step to tag these nodes as controllers with the osprole=controller property.

heat-admin@controller-1 $ for controller in ${controllers}; do sudo pcs property set --node ${controller} osprole=controller ; done
heat-admin@controller-1 $ sudo pcs property

The newly assigned roles should appear in the Node attributes section.

Create a list of STONITH devices that are already present in the environment.

heat-admin@controller-1 $ STONITHdevs=$(sudo pcs stonith | awk '{print $1}')
heat-admin@controller-1 $ echo $stonithdevs

Tag the control plane services to make sure they only run on listed Controller nodes and skip any listed STONITH devices.

heat-admin@controller-1 $ for i in $(sudo cibadmin -Q --xpath //primitive --node-path | tr ' ' '\n' | awk -F "id='" '{print $2}' | awk -F "'" '{print $1}' | uniq); do
    found=0
    if [ -n "$stonithdevs" ]; then
        for x in $stonithdevs; do
            if [ $x = $i ]; then
                found=1
            fi
	    done
    fi
    if [ $found = 0 ]; then
        sudo pcs constraint location $i rule resource-discovery=exclusive score=0 osprole eq controller
    fi
done

Set the requires property to fencing as the default for all resources. This operation is required to enable the unfence mechanism.

heat-admin@controller-1 $ sudo pcs resource defaults requires=fencing

Create a separate fence-nova STONITH device.

heat-admin@controller-1 $ . overcloudrc.v3
heat-admin@controller-1 $ sudo pcs stonith create fence-nova fence_compute \
                                auth-url=$OS_AUTH_URL \
                                login=$OS_USERNAME \
                                passwd=$OS_PASSWORD \
                                tenant-name=$OS_TENANT_NAME \
                                domain=localdomain record-only=1 \
                                meta provides=unfencing \
                                --force

Create a pacemaker constraint for fence-nova to restrict the service to Controller nodes and set resource-discovery to never.

heat-admin@controller-1 $ pcs constraint location fence-nova rule resource-discovery=never score=0 osprole eq controller

Add a STONITH device on each Compute nodes. This command should be run separately on each Compute node.

heat-admin@controller-1 $ sudo pcs stonith create ipmilan-overcloud-compute-N  fence_ipmilan pcmk_host_list=overcloud-compute-N ipaddr=IPADDR login=IPMILANUSER passwd=IPMILANPW lanplus=1 cipher=1 op monitor interval=60s;

Where:

Make sure that the Compute nodes can recover after fencing.

heat-admin@controller-1 $ sudo pcs property set cluster-recheck-interval=1min

Create a pacemaker remote resource for each Compute node and set osprole to compute.

heat-admin@controller-1 $ sudo pcs resource create overcloud-compute-n ocf:pacemaker:remote reconnect_interval=240 op monitor interval=20"
heat-admin@controller-1 $ sudo pcs property set --node overcloud-compute-n osprole=compute"

Create Compute node resources and set the STONITH level 1 to include the physical fence device of the nodes and the fence-nova service. Run this command separately on each Compute node.

heat-admin@controller-1 $ sudo pcs resource create overcloud-compute-N ocf:pacemaker:remote reconnect_interval=60 op monitor interval=20
heat-admin@controller-1 $ sudo pcs property set --node overcloud-compute-N osprole=compute
heat-admin@controller-1 $ sudo pcs stonith level add 1 overcloud-compute-N ipmilan-overcloud-compute-N,fence-nova
heat-admin@controller-1 $ sudo pcs stonith

Replace N with the identifying number of each Compute node, for example, overcloud-compute-1, overcloud-compute-2, and so on. Use these identifying numbers to match each Compute node with the STONITH devices that you created earlier, for example, match overcloud-compute-1 with ipmilan-overcloud-compute-1.

After you complete this procedure, allow some time for the environment to settle and then cleanup any failed resources.

heat-admin@controller-1 $ sleep 60
heat-admin@controller-1 $ sudo pcs resource cleanup
heat-admin@controller-1 $ sudo pcs status
heat-admin@controller-1 $ sudo pcs property set stonith-enabled=true