fence_xvm fails when called by stonith despite it working on the command line and fence_virt on the host seeing the fence requests in RHEL 6 or 7 pacemaker clusters

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6 or 7 with the High Availability Add On
  • One or more stonith devices configured to use fence_xvm or fence_virt
  • VMs using node names that do not match their name on the virtualization host

Issue

  • I can't seem to get fence_xvm to work
  • When I create a fence_xvm stonith device in my cluster, it works from the command line but always fails when a node needs to be fenced.
  • fence_virt fails with unknown errors even though everything is set up correctly
  • pcs status shows "unknown error" for my fence_xvm stonith device even though it works properly from the command line
# pcs status
[...]
Full list of resources:

virtfence (stonith:fence_xvm):   Stopped

Failed actions:
  virtfence_start_0 on node1.example.com 'unknown error' (1): call=41, status=Error, last-rc-change='Thu May 15 14:48:06 2014', queued=7014ms, exec=0ms
  virtfence_start_0 on node2.example.com 'unknown error' (1): call=41, status=Error, last-rc-change='Thu May 15 14:48:14 2014', queued=7012ms, exec=0ms

Resolution

Specify a pcmk_host_map to map the node-names in the cluster to their real names on the virtualization host.

Root Cause

When a stonith device is created with the default setting of pcmk_host_check=dynamic-list, stonith will automatically pass in the name of node to fence as the port attribute. However, in the case of virtualization-based agents, it is common for the node name to not match the name of the VM on the virtualization host, and thus this port attribute ends up being wrong, causing fence attempts to fail. By specifying a pcmk_host_map, one can tell stonith to pass a different port value for each node.

Diagnostic Steps

  • Check communications between the nodes and the virtualization host, to ensure there is not something blocking requests.

  • Run fence_virtd on the virtualization host to see if requests are actually arriving. If not, the configuration may be incorrect or a network problem may exist.

  • Attempt to run fence_xvm or fence_virt manually from the command line on one or more cluster nodes, specifying the appropriate parameters. If this succeeds but fails from stonith when configured with the same parameters, then the issue described above is a good candidate for the cause of this problem.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.