New configuration is not applied by rgmanager or displayed by clustat after fencing failed at some point in the past in RHEL 5

Solution In Progress - Updated -

Issue

  • clustat doesn't show changes applied to cluster.conf on all nodes
  • rgmanager is not applying configuration changes
  • No cluster configuration refresh after the creation of a new resource.
  • We have created a new resource and updated the configuration with the command ccs_tool update /etc/cluster/cluster.conf, but after the update we only see the new resource in the node where we executed the command.
  • We added a service in /etc/cluster/cluster.conf, incremented the cluster version number, did a ccs_tool update /etc/cluster/cluster.conf...After this the cluster.conf was updated on both nodes (checked with cman_tool status - ok); but the service is not recognized on one node and the clurgmgrd does not check any more the services.
  • A node rebooted and a node outside the failoverdomain took it over. When the node came back into the cluster, the service did not failback to either of the higher priority nodes in the failoverdomain, but stayed outside the domain. It was later discovered from an rgmanager-debug dump that the node which took over the service was still using an old config version which had different failoverdomains, even though cman showed as using the latest version.

Environment

  • Red Hat Enterprise Linux (RHEL) 5 with the High Availability Add On
  • rgmanager prior to release 2.0.52-47.el5
  • One or more nodes required fencing at some point in the past, fencing failed, and the node eventually rejoined the cluster without being fenced
    • rgmanager was never restarted on one or more nodes since that rejoin
  • The configuration version reported by cman and rgmanager (clurgmgrd) are different (see Diagnostic Steps)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content