Cman binds to wrong ip address when 2 NICs on subnet with Red Hat High Availability Addon

Solution In Progress - Updated -

Issue

  • After rebooting both cluster nodes, CMAN failed to start and throw "TOTEM" error. I can ping cluster IP successfully but cluster is showing some network error in "UNICAST" mode. When I changed it to "MULTICAST" mode, CMAN successfully starts but created a fencing loop where each server reboots one after another comes live.

Environment

  • Red Hat Enterprise Linux 6 (With High Availability or Resilient Storage Add-ons)
  • High Availability Cluster with 2 or more nodes
  • 2 or more network interfaces active on cluster nodes

    • At least 2 network interfaces are on the same subnet. For example, in the following example, eth1 and bond0 are both on 10.0.0.0/24:

      $ grep -B2 -e 10.0.0.XX -e 10.0.0.YY sos_commands/networking/ip_address 
      3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
          link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
          inet 10.0.0.YY/24 brd 10.0.0.255 scope global eth1
      --
      6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
          link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
          inet 10.0.0.XX/24 brd 10.0.0.255 scope global bond0
      
    • One of the interfaces on the same subnet is configured to be used for cluster communication:

      # cman_tool status | grep "Node addresses"
      Node addresses: 10.0.0.YY
      

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content