overcloud global response times are slower when 1 controller is offline

Issue

• We have just re-installed our overcloud with RHOSP13 Z8.

• Whenever we take one controller node offline we see a jump in the amount of time it takes to run openstack commands.

• Whenever we take one controller in the cluster offline using following commands:

# pcs cluster stop
# shutdown now -P

• The following table explains the situation quite eloquently:
  

  Action	execution time with all controllers online	execution time with one controller down
  Volume Create x 1	real 0m2.911s user 0m1.488s sys 0m0.316s	real 0m2.214s user 0m1.401s sys 0m0.336s
  Volume List	real 0m2.732s user 0m1.581s sys 0m0.380s	real 0m2.393s user 0m1.530s sys 0m0.354s
  Instance Create x 1	real 0m2.794s user 0m1.396s sys 0m0.249s	real 0m33.635s user 0m1.467s sys 0m0.313s