One cluster node gets fenced after running `firewall-cmd --reload`

Solution In Progress - Updated -

Issue

  • Reloading firewalld in a cluster with knet transport causes a node to be fenced.
[root@rhel-8-0-2 ~]# date && time firewall-cmd --reload && date
Mon Jan  7 01:10:13 PST 2019
success

real    0m1.757s
user    0m0.266s
sys 0m0.036s
Mon Jan  7 01:10:15 PST 2019

[root@rhel-8-0-2 ~]# cat /var/log/messages
...
Jan  7 01:10:14 rhel-8-0-2 corosync[822]:  [KNET  ] link: host: 1 link: 0 is down
Jan  7 01:10:14 rhel-8-0-2 corosync[822]:  [TOTEM ] Token has not been received in 750 ms
Jan  7 01:10:14 rhel-8-0-2 corosync[822]:  [TOTEM ] A processor failed, forming new configuration.
Jan  7 01:10:16 rhel-8-0-2 corosync[822]:  [TOTEM ] A new membership (2:6004) was formed. Members left: 1
Jan  7 01:10:16 rhel-8-0-2 corosync[822]:  [TOTEM ] Failed to receive the leave message. failed: 1
Jan  7 01:10:16 rhel-8-0-2 corosync[822]:  [CPG   ] downlist left_list: 1 received
Jan  7 01:10:16 rhel-8-0-2 corosync[822]:  [QUORUM] Members[1]: 2
Jan  7 01:10:16 rhel-8-0-2 corosync[822]:  [MAIN  ] Completed service synchronization, ready to provide service.
Jan  7 01:10:16 rhel-8-0-2 pacemaker-controld[842]: notice: Node rhel-8-0-1 state is now lost
Jan  7 01:10:16 rhel-8-0-2 pacemaker-fenced[838]: notice: Node rhel-8-0-1 state is now lost
Jan  7 01:10:16 rhel-8-0-2 pacemaker-based[837]: notice: Node rhel-8-0-1 state is now lost
Jan  7 01:10:16 rhel-8-0-2 pacemaker-controld[842]: warning: Our DC node (rhel-8-0-1) left the cluster
Jan  7 01:10:16 rhel-8-0-2 pacemaker-based[837]: notice: Purged 1 peer with id=1 and/or uname=rhel-8-0-1 from the membership cache
Jan  7 01:10:16 rhel-8-0-2 pacemakerd[836]: notice: Node rhel-8-0-1 state is now lost
Jan  7 01:10:16 rhel-8-0-2 pacemaker-fenced[838]: notice: Purged 1 peer with id=1 and/or uname=rhel-8-0-1 from the membership cache
Jan  7 01:10:16 rhel-8-0-2 pacemaker-controld[842]: notice: State transition S_NOT_DC -> S_ELECTION
Jan  7 01:10:16 rhel-8-0-2 pacemaker-controld[842]: notice: State transition S_ELECTION -> S_INTEGRATION
Jan  7 01:10:16 rhel-8-0-2 pacemaker-attrd[840]: notice: Node rhel-8-0-1 state is now lost
Jan  7 01:10:16 rhel-8-0-2 pacemaker-attrd[840]: notice: Removing all rhel-8-0-1 attributes for peer loss
Jan  7 01:10:16 rhel-8-0-2 pacemaker-attrd[840]: notice: Purged 1 peer with id=1 and/or uname=rhel-8-0-1 from the membership cache
Jan  7 01:10:16 rhel-8-0-2 pacemaker-schedulerd[841]: warning: Cluster node rhel-8-0-1 will be fenced: peer is no longer part of the cluster
Jan  7 01:10:16 rhel-8-0-2 pacemaker-schedulerd[841]: warning: Node rhel-8-0-1 is unclean

Environment

  • Red Hat Enterprise Linux (RHEL) 8 with the High Availability Add-on
  • knet transport

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In