One or more nodes is killed in a RHEL 5 cluster after nodes report "openais[xxxx]: Killing node nodeX because it has rejoined the cluster with existing state"

Issue

• Both nodes in the cluster killed each other simultaneously after a token loss:

  openais[3707]: [MAIN ] Killing node node2.example.com because it has rejoined the cluster with existing state 
  openais[3707]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
  gfs_controld[2759]: cluster is down, exiting
  dlm_controld[2753]: cluster is down, exiting
  kernel: dlm: closing connection to node 2
  

• Why did my two node cluster go down after a network failure?

• There was a network split in my cluster, followed by the network then recovering, and a node was killed because it "rejoined the cluster without a full restart".

  Nov  6 23:30:20 node1 openais[13068]: [MAIN ] Killing node node2 because it has rejoined the cluster with existing state 
  Nov  6 23:30:20 node1 openais[13068]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart 
  
  Nov  6 23:30:20 node2 openais[13382]: [MAIN ] Killing node node1 because it has rejoined the cluster with existing state 
  Nov  6 23:30:20 node2 openais[13382]: [CMAN ] cman killed by node 1 because we rejoined the cluster without a full restart 
  

• service cman status command shows "groupd dead but pid file exists", and cluster could not be formed:

  # service cman status
  groupd dead but pid file exists
  

• Have issue with two node Red Hat High Availability Cluster, both machines are going shutdown by fencing each other.

• Red Hat Cluster suit issue, we found cluster is not quorate and refusing connection. Cluster servers fence all the nodes, what causing the cluster fence each other?