RHEL 7 High Availability node unable to join cluster, corosync reports "[QB ] Denied connection, is not ready", and reports continuous "Retransmit List"
Issue
- A cluster which was previously working for long time, suddenly starts missbehaving
- I have several nodes in the cluster that are able to join and communicate just fine, but one reports the others as "unclean" in
pcs statusoutput, and it shows constant "[TOTEM] Retransmit List" messages and "[QB ] Denied connection, is not ready" messages over and over in the logs
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.1.1}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.254}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.253}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] adding new UDPU member {192.10.255.252}
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] A new membership (192.10.255.252:4) was formed. Members joined: 4
Sep 15 22:34:55 node4 corosync[56061]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 6
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] A new membership (192.10.1.1:12) was formed. Members joined: 1 5 6 3 2
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
[...]
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56065-18)
Sep 15 22:34:55 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:55 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56067-19)
Sep 15 22:34:56 node4 corosync[56061]: [TOTEM ] Retransmit List: 3 4 5 a
Sep 15 22:34:56 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56069-20)
Sep 15 22:34:56 node4 corosync[56061]: [QB ] Denied connection, is not ready (56061-56071-21)
Environment
- Red Hat Enterprise Linux (RHEL) 7 with the High Availability Add On
totem { transport: udpu }in/etc/corosync/corosync.conf
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.