"[TOTEM] Retransmit List" messages repeatedly seen in RHEL 5, 6, or 7 High Availability cluster node logs

Updated 2015-03-09T15:53:36+00:00

Issue

  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red hat Enterprise Linux 5 cluster nodes:
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053c6  
Jul 25 04:20:16 node1 openais[18311]: [TOTEM] Retransmit List: 6053d0  
  • There are lots of repeating messages written to /var/log/messages similar to the following below on Red Hat Enterprise Linux 6 cluster nodes:
Sep 22 12:37:45 node1 corosync [TOTEM] Retransmit List: 31 
Sep 22 12:37:46 node1 corosync [TOTEM] Retransmit List 1
  • RHEL nodes fails to join the cluster with Retransmit List errors in logs.
  • clvmd hangs and lvm commands like lvs, lvdisplay, etc do not respond after seeing repeating "Retransmit List" messages in the logs.
  • rgmanager or clusvcadm operations do not complete when there are retransmits in /var/log/messages.
  • GFS2 I/O hangs for short periods and the logs show "Retransmit List" repeating over and over during that time.

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, or 7 with the High Availability or Resilient Storage Add On
  • openais or corosync based clusters

Subscriber content preview. For full access to the Red Hat Knowledgebase, please log in.

Not a subscriber? Learn more about the benefits of Red Hat Subscriptions.