"[TOTEM] Retransmit List" messages repeatedly seen in RHEL 5, 6, or 7 High Availability cluster node logs

Solution Verified - Updated -

Issue

  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red hat Enterprise Linux 5 cluster nodes:
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053c6  
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053d0  
  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red Hat Enterprise Linux 6 cluster nodes:
Sep 22 12:37:45 node1 corosync [TOTEM] Retransmit List: 31 
Sep 22 12:37:46 node1 corosync [TOTEM] Retransmit List 1
  • RHEL nodes fails to join the cluster with Retransmit List errors in logs.
  • clvmd hangs and lvm commands like lvs, lvdisplay, etc do not respond after seeing repeating "Retransmit List" messages in the logs.
  • rgmanager or clusvcadm operations do not complete when there are retransmits in /var/log/messages.
  • GFS2 I/O hangs for short periods and the logs show "Retransmit List" repeating over and over during that time.
  • Cluster node hangs at ''Joining fence domain'' while booting or while starting cman service after normal boot.
  • Getting openais retransmission errors in logs on cluster nodes.

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the High Availability or Resilient Storage Add On
  • openais or corosync based clusters

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.