RHEL 7 High Availability node unable to join cluster, corosync reports "[QB ] Denied connection, is not ready", and reports continuous "Retransmit List"
Issue
- A cluster which was previously working for long time, suddenly starts missbehaving
- I have several nodes in the cluster that are able to join and communicate just fine, but one reports the others as "unclean" in
pcs statusoutput, and it shows constant "[TOTEM] Retransmit List" messages and "[QB ] Denied connection, is not ready" messages over and over in the logs
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.1.1}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.254}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.253}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.252}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] A new membership (192.10.255.252:4) was formed. Members joined: 4
Sep 15 22:34:55 node4 corosync[56061]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 6
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] A new membership (192.10.1.1:12) was formed. Members joined: 1 5 6 3 2
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
[...]
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56065-18)
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56067-19)
Sep 15 22:34:56 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:56 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56069-20)
Sep 15 22:34:56 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56071-21)
Environment
- Red Hat Enterprise Linux (RHEL) 7 with the High Availability Add On
totem { transport: udpu }in/etc/corosync/corosync.conf
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
