Node that loses quorum strips a tag for an lvm resource it doesn't own before it is fenced and next status check for that resource fails on the service owner with "<err> WARNING: <node> does not own <vg>" in a RHEL 6 High Availability cluster

Solution Unverified - Updated -

Issue

  • A node had a network issue but before it was fenced seems to have stripped the tag for an lvm resource that it didn't own:
Nov  7 11:15:32 rhel6-node2 corosync[2352]:   [QUORUM] Members[1]: 2
Nov  7 11:15:32 rhel6-node2 corosync[2352]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Nov  7 11:15:32 rhel6-node2 rgmanager[2578]: #1: Quorum Dissolved
Nov  7 11:15:32 rhel6-node2 kernel: dlm: closing connection to node 1
Nov  7 11:15:32 rhel6-node2 corosync[2352]:   [CPG   ] chosen downlist: sender r(0) ip(192.168.143.62) ; members(old:3 left:2)
Nov  7 11:15:32 rhel6-node2 corosync[2352]:   [MAIN  ] Completed service synchronization, ready to provide service.
Nov  7 11:15:32 rhel6-node2 kernel: dlm: closing connection to node 3
[...]
Nov  7 11:15:36 rhel6-node2 rgmanager[5356]: [lvm] Stripping tag, rhel6-node1.example.com
  • After one node split from the cluster and was fenced, a status check for an lvm resource failed on another node, reporting in the logs:
[lvm] WARNING: clust should not be active
<err>    WARNING: rhel6-node1.example.com does not own clust
  • Using HA LVM tagging, cluster experiences incorrect vg tag clearing and uninvolved cluster services recover if a node fails and if post-fail-delay is long.

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add On
  • rgmanager
  • resource-agents releases prior to 3.9.5-12.el6_6.2
  • Cluster with 3 or more nodes, or a two-node cluster with a quroum device (any cluster where one node can lose quorum)
  • HA-LVM with the tagging variant
    • <lvm/> resource does not have an lv_name specified
  • This issue is more likely in environments with a delay before a node is fenced, such as those with a post_fail_delay, a fencedevice delay, or a slow fencing method.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content