"[TOTEM] Retransmit List" messages repeatedly seen in RHEL 5, 6, or 7 High Availability cluster node logs

Solution Verified - Updated -

Issue

  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red hat Enterprise Linux 5 cluster nodes:
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053c6  
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053d0  
  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red Hat Enterprise Linux 6 cluster nodes:
Sep 22 12:37:45 node1 corosync [TOTEM] Retransmit List: 31 
Sep 22 12:37:46 node1 corosync [TOTEM] Retransmit List 1
  • RHEL nodes fails to join the cluster with Retransmit List errors in logs.
  • clvmd hangs and lvm commands like lvs, lvdisplay, etc do not respond after seeing repeating "Retransmit List" messages in the logs.
  • rgmanager or clusvcadm operations do not complete when there are retransmits in /var/log/messages.
  • GFS2 I/O hangs for short periods and the logs show "Retransmit List" repeating over and over during that time.
  • Cluster node hangs at ''Joining fence domain'' while booting or while starting cman service after normal boot.
  • Getting openais retransmission errors in logs on cluster nodes.

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the High Availability or Resilient Storage Add On
  • openais or corosync based clusters

Subscriber exclusive content

A Red Hat Subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In