A failed or unresponsive node is never fenced after corosync reports "A processor failed, forming new configuration" while one or more nodes is repeatedly reporting "Unable to load new config in corosync" in a RHEL 6 High Availability cluster

Solution Unverified - Updated -

Issue

  • After a node became unresponsive, why didn't the cluster fence it? It looks like we did see a token loss on all nodes, but we don't see a fence here.
Nov 17 02:24:01 node1 corosync[59868]:   [TOTEM ] A processor failed, forming new configuration.
Nov 17 02:25:43 node1 corosync[59868]:   [QUORUM] Members[5]: 2 3 4 5 6
Nov 17 02:25:43 node1 corosync[59868]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Nov 17 02:25:43 node1 corosync[59868]:   [CPG   ] chosen downlist: sender r(0) ip(10.204.3.16) ; members(old:6 left:1)
Nov 17 02:25:43 node1 corosync[59868]:   [MAIN  ] Completed service synchronization, ready to provide service.
  • A node failed, and the rest of the cluster nodes all reported "a processor failed", but none of them fenced the missing node or reported "fencing deferred to ".
  • I see one node repeating over and over that its "Unable to load new config in corosync", and while this is ongoing a node was removed from the cluster but didn't get fenced
Nov 17 02:01:54 node3 corosync[47928]:   [CMAN  ] Unable to load new config in corosync: New configuration version has to be newer than current running configuration
Nov 17 02:01:54 node3 corosync[47928]:   [CMAN  ] Can't get updated config version 328: New configuration version has to be newer than current running configuration#012.
Nov 17 02:01:54 node3 corosync[47928]:   [CMAN  ] Activity suspended on this node
Nov 17 02:01:54 node3 corosync[47928]:   [CMAN  ] Error reloading the configuration, will retry every second
  • After I updated the config, fencing no longer happens if a node fails.

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.