RHEL 7 High Availability node unable to join cluster, corosync reports "[QB ] Denied connection, is not ready", and reports continuous "Retransmit List"

Solution Unverified - Updated -

Issue

  • A cluster which was previously working for long time, suddenly starts missbehaving
  • I have several nodes in the cluster that are able to join and communicate just fine, but one reports the others as "unclean" in pcs status output, and it shows constant "[TOTEM] Retransmit List" messages and "[QB ] Denied connection, is not ready" messages over and over in the logs
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] adding new UDPU member {192.10.1.1}
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] adding new UDPU member {192.10.255.254}
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] adding new UDPU member {192.10.255.253}
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] adding new UDPU member {192.10.255.252}
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] A new membership (192.10.255.252:4) was formed. Members joined: 4
Sep 15 22:34:55 node4 corosync[56061]:  [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 6
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] A new membership (192.10.1.1:12) was formed. Members joined: 1 5 6 3 2
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
[...]
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [QB    ] Denied connection, is not ready (56061-56065-18)
Sep 15 22:34:55 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:55 node4 corosync[56061]:  [QB    ] Denied connection, is not ready (56061-56067-19)
Sep 15 22:34:56 node4 corosync[56061]:  [TOTEM ] Retransmit List: 3 4 5 a 
Sep 15 22:34:56 node4 corosync[56061]:  [QB    ] Denied connection, is not ready (56061-56069-20)
Sep 15 22:34:56 node4 corosync[56061]:  [QB    ] Denied connection, is not ready (56061-56071-21)

Environment

  • Red Hat Enterprise Linux (RHEL) 7 with the High Availability Add On
  • totem { transport: udpu } in /etc/corosync/corosync.conf

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content