Why does my RHEL High Availability cluster form but fail after a few minutes, or why do my multicast communications stop working after a short amount of time?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the High Availability Add-On

Issue

  • My cluster works for a short time and then after a few minutes everything stops working suddenly.
  • My cluster will not form a quorate cluster, it fails and /var/log/messages show Retransmit List messages logged sometime.
  • My cluster forms but fails shortly after, or fails when I mount gfs2 volumes or failover a service
  • "Retransmit List" messages are seen frequently on my cluster, with the entries in the list never being removed and the list continuing to grow:
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 62 
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 62 63 
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 62 63 64
  • When attempting to change the state of a cluster service, a long string of "Retransmit List" messages is seen and the operation fails with "Failed changing RG status":
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 62 63 64 65 66
Jun 10 14:46:24 node1 corosync[3266]:   [TOTEM ] Retransmit List: 5e 5f 60 61 62 63 64 65 66
Jun 10 14:46:24 node1 rgmanager[4241]: #55: Failed changing RG status
  • It seems like the cluster has multicast issues, but when I test multicast it seems to work fine.
  • I created a pacemaker cluster, with three members. My problem is that one of the members is no longer seeing the others online. It thus assumes a cluster split, and as it doesn't have quorum to start the resources?
  • As part of testing cluster functionalities, We have powered off the one server in the 2 node cluster. The server which was powered off , it came up automatically which I understand that fencing account has reset the server to come up. But when the server came up, other node also got rebooted and it caused outage for application. What could be the cause of this issue?
  • When I create 2 node cluster (without quorum) on RHEL6.5, I can set it up properly, but after rebooting the server for sometimes, the cluster nodes are not clustered together and each node forms a standalone cluster where clustat shows the self as Online and other node as offline.

Resolution

  • Many switches come with IGMP snooping enabled by default. This configuration requires an IGMP querier and PIM router or forwarder to be configured on the subnet to manage IGMP groups and traffic and keep routing tables up-to-date with the latest group membership. Configuring these components varies on different hardware and network layouts, so it is recommended that you consult with your network vendor to determine the appropriate method for enabling multicast traffic. A reference is available for Cisco switches.

  • Ensure that the network and subnet configuration for the nodes is applicable for the network they are on and allows them to reach the IGMP querier. For instance, if the nodes are on a VLAN using a /24 subnet (255.255.255.0) and configured with an IGMP querier, but the nodes' interfaces are configured to use a /29 subnet, it may put the querier outside the addressable range for them.

  • In some cases, it may make sense to disable IGMP snooping on the switch(es) used by the cluster, which results in them no longer maintaining multicast membership tables and instead forwards the traffic to all other ports. This can have a negative impact on network performance and can potentially lead to other problems, so it is typically not recommended. Consult with your network vendor for assistance in determining if this option is appropriate.

Root Cause

  • With IGMP snooping enabled a switch will maintain a table of multicast group members. When the cluster starts, each node sends a request to join this group, and the switch creates the table. After some period of time, the table entries are marked as stale and are dropped. Normally an IGMP querier should be configured in conjunction with IGMP snooping, as this querier will frequently update the membership tables on the different switches, preventing them from being aged out. When a querier is not configured, nodes in the cluster may stop receiving multicast messages which can lead to "Retransmit List" messages, membership changes, fence events, and service operation failures.

  • Refer to this article for a more detailed explanation of the communication protocol used by RHEL 5 and 6 clusters.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments