Cluster fails to form membership when totem token is set to 30s or longer

Issue

Cluster nodes are able to form membership if the totem token has value up to 29000ms. Once the totem token is set to value 30000ms or more the cluster nodes fail to establish a connection between the nodes and therefore staying inquorate.

Mar 10 11:46:05 node1 corosync[40837]:  [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 6
Mar 10 11:46:05 node1 corosync[40837]:  [QUORUM] Members[1]: 2
Mar 10 11:46:05 node1 corosync[40837]:  [MAIN  ] Completed service synchronization, ready to provide service. 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 has no active links 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 systemd[1]: Started Corosync Cluster Engine.
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 0) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 1 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 3 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 4 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 5 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1) 
Mar 10 11:46:05 node1 corosync[40837]:  [KNET  ] host: host: 6 has no active links
Mar 10 11:46:05 node1 pacemakerd[40850]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log
Mar 10 11:46:05 node1 pacemakerd[40850]: notice: Initiated blackbox recorder: /var/lib/pacemaker/blackbox/pacemakerd-40850
Mar 10 11:46:05 node1 pacemakerd[40850]: notice: Starting Pacemaker 2.0.3-5.el8_2.3
Mar 10 11:46:05 node1 pacemakerd[40850]: warning: Quorum lost