Skip to navigation

Red Hat Cluster Suite nodes will not join when using Cisco network switches on Red Hat Enterprise Linux 5

Updated 2013-12-29T20:00:01+00:00

Issue

Problem

  • Nodes are online and communicating but can't form a quorate cluster
  • Nodes start cman but take several minutes to detect the other node via multicast (openais):
    
    Node1:
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ] CLM CONFIGURATION CHANGE
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ] New Configuration:
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ]  r(0) ip(10.16.177.21)
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ] Members Left:
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ] Members Joined:
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ]  r(0) ip(10.16.177.21)
    Oct 19 04:34:41 node01 openais[4000]: [SYNC ] This node is within the primary component and will provide service.
    Oct 19 04:34:41 node01 openais[4000]: [TOTEM] entering OPERATIONAL state.
    Oct 19 04:34:41 node01 openais[4000]: [CMAN ] quorum regained, resuming activity
    Oct 19 04:34:41 node01 openais[4000]: [CLM  ] got nodejoin message 10.16.177.21
    
    Node2:
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ] CLM CONFIGURATION CHANGE 
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ] New Configuration: 
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ]  r(0) ip(10.16.177.22)  
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ] Members Left: 
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ] Members Joined: 
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ]  r(0) ip(10.16.177.22)  
    Oct 19 04:34:45 node02 openais[3365]: [SYNC ] This node is within the primary component and will provide service. 
    Oct 19 04:34:45 node02 openais[3365]: [TOTEM] entering OPERATIONAL state. 
    Oct 19 04:34:45 node02 openais[3365]: [CMAN ] quorum regained, resuming activity 
    Oct 19 04:34:45 node02 openais[3365]: [CLM  ] got nodejoin message 10.16.177.22 
    Then 3 minutes later the nodes see each other:
    
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ] CLM CONFIGURATION CHANGE
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ] New Configuration:
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ]  r(0) ip(10.16.177.21)
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ]  r(0) ip(10.16.177.22)
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ] Members Left:
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ] Members Joined:
    Oct 19 04:37:55 node01 openais[4000]: [CLM  ]  r(0) ip(10.16.177.22)
    Oct 19 04:37:55 node01 openais[4000]: [SYNC ] This node is within the primary component and will provide service.
    Oct 19 04:37:55 node01 openais[4000]: [TOTEM] entering OPERATIONAL state.
    Oct 19 04:37:55 node01 openais[4000]: [MAIN ] Killing node node02 because it has rejoined the cluster with existing state
    Oct 19 04:37:55 node01 openais[4000]: [CMAN ] cman killed by node 2 because we rejoined the cluster without a full restart
    
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ] CLM CONFIGURATION CHANGE
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ] New Configuration:
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ]  r(0) ip(10.16.177.21)
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ]  r(0) ip(10.16.177.22)
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ] Members Left:
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ] Members Joined:
    Oct 19 04:37:55 node02 openais[3365]: [CLM  ]  r(0) ip(10.16.177.21)
    Oct 19 04:37:55 node02 openais[3365]: [SYNC ] This node is within the primary component and will provide service.
    Oct 19 04:37:55 node02 openais[3365]: [TOTEM] entering OPERATIONAL state.
    Oct 19 04:37:55 node02 openais[3365]: [MAIN ] Killing node node01 because it has rejoined the cluster with existing state
    Oct 19 04:37:55 node02 openais[3365]: [CMAN ] cman killed by node 1 because we rejoined the cluster without a full restart
    

Environment

  • Red Hat Enterprise Linux (RHEL) 5.3
  • Red Hat Cluster Suite (RHCS)
  • Network used for cluster communication contains a Cisco switch
  • Network switches haven't been configured to allow multicast in the network range used by openais in Red Hat Cluster Suite.

Subscriber content preview. For full access to the Red Hat Knowledgebase, please log in.

Not a subscriber? Learn more about the benefits of Red Hat Subscriptions.