Support Policies for RHEL High Availability Clusters - fence_scsi and fence_mpath

Contents

• Overview
  • Applicable Environments
  • Useful References and Guides
  • Introduction
• Policies

Overview

Applicable Environments

• Red Hat Enterprise Linux (RHEL) with the High Availability Add-On

Useful References and Guides

• Support Policies for RHEL High Availability Clusters

Introduction

This guide offers Red Hat's policies and requirements around the usage of fence_scsi or fence_mpath in a RHEL High Availability cluster. These agents are similar in their implementation and functionality, so they generally follow the same policies and have similar requirements. Users of RHEL High Availability clusters should adhere to these policies in order to be eligible for support from Red Hat with the appropriate product support subscriptions.

Policies

Storage use-cases for fence_scsi and fence_mpath

These STONITH methods which use SCSI persistent reservations are only valid fencing solutions in clusters where all managed applications utilize data from storage devices shared by all members.

fence_scsi and fence_mpath control cluster members' access to shared storage devices based on their membership status and resource management results. If a problem occurs with a member, the cluster will revoke access to the storage devices for that member, so that other members can safely interact with managed applications and resources without risk of conflict. This method is only effective if loss of access to storage will prevent that problematic member from carrying out conflicting actions with an application or resource.

Red Hat's support stance for these agents is thus:

• The fence-agents fence_scsi and fence_mpath only satisfy a cluster's requirement to have a valid fence device IF all managed applications are designed to interact with one or more shared storage devices. Presenting a dedicated storage device whose only purpose is to serve as a fencing mechanism does NOT meet the requirements for these agents. The agent must be configured to manage devices that serve data to the managed application(s) in the cluster - a dedicated fence storage device does not suffice.
• All storage devices which are shared by any members must be shared by ALL members.
• The fence-agents fence_scsi and fence_mpath is only supported when it is used (-d <storage devices>, devices=<storage devices) with whole storage devices (ex. /dev/sdf, /dev/mapper/mpath_k, etc) and is not supported on partitions (ex. /dev/sdf1, /dev/mapper/mpath_k1, etc) of a storage device.
• The fence-agents fence_scsi and fence_mpath must be configured to manage ALL shared storage devices (see next section).

STONITH device must manage all shared storage

In order for fence_scsi or fence_mpath to serve its intended purpose, and in order for the configuration to be supported by Red Hat, it must manage all storage devices that are shared by more than one member of the cluster.

All shared storage devices should be included in devices section or autodetect by fence_scsi or fence_mpath because if they are not then shared storage device can get corrupted. Red Hat will not assist with concerns arising out of configurations that do not properly manage all shared storage devices in the cluster.

• fence_scsi and fence_mpath are capable of either auto-detecting the device list or accepting an administrator-specified device list in its parameters.
  • If devices is a specified parameter in the configuration of the STONITH/fence device, its value must include all shared storage devices in the cluster.
  • If devices is not a specified parameter in the configuration of the STONITH/fence device, then all shared storage devices in the cluster must be physical volumes in LVM volume groups marked as "clustered" (with the 'c' LVM attribute).
• There has to be at least one resource (usually it will be a LVM-activate resource and/or Filesystem resource) managed by pacemaker that requires access to the shared storage device in order to operate. For example an unsupported configuration would be the following:
  • A pacemaker cluster that uses fence_scsi or fence_mpath as the stonith device, and a single IPaddr2 resource. This cluster would be unsupported as there is no shared storage device used by pacemaker and thus any fence action taken would not affect the status of the resource. For more information on this then see: Available Fencing Types and Fencing Agents for a Red Hat High-Availability Cluster | storage fencing.

Shared storage on device mapper multipath devices: Our recommendation is that fence_mpath should be used as the fence-agent whenever the shared storage devices use device mapper multipath (dmm) managed storage . The fence-agent fence_scsi can still be used on dmm devices, but should use the path to the dmm device file instead of the path to the physical device.

SPC-3 required

It is required that the storage managed by fence_scsi or fence_mpath is compliant with the SCSI Primary Commands version 3 (SPC-3) standard, in order for assistance to be provided by Red Hat.

fence_mpath with device-mapper-multipath

fence_mpath provides SCSI persistent reservation fencing for devices that are managed by device-mapper-multipath. This agent is not supported with any other multipath technology, or when device-mapper-multipath is not in use.

• fence_mpath requires that /etc/multipath.conf be configured with a unique reservation_key hexadecimal value on each node, either in the defaults or in a multipath block for each cluster-shared device. For example:

multipaths {
    multipath {
            wwid "1234567890" # wwid for a cluster-shared device
            alias "ha_data1"
            reservation_key 0x1 # unique to this node - change for other nodes
    }
}

Storage compatibility

Red Hat's typical policies with respect to storage compatibility for RHEL High Availability apply to the usage of fence_scsi or fence_mpath, in that Red Hat does not test, guarantee, or certify successful operation of this agent with specific storage vendor products. It is important that customer organizations seeking to use these agents test the operation fully with the technologies that will work together in the target cluster environment, and work with their storage vendors and Red Hat Support if any unexpected behaviors arise.

• Red Hat's target in testing these agents is general SPC-3 compliant storage that supports SCSI persistent reservations, and will provide support for usage of fence_scsi and fence_mpath with storage that adheres to these relevant standards - unless otherwise stated in Red Hat's support policies.
• Special storage technologies such as replication, multiple-site or multiple array coordination, or other vendor-specific features are not tested and could affect the behavior of these agents.
• See also: Support policies - storage compatibility

Storage replication

Use of storage replication can present challenges with fence_scsi or fence_mpath, in that different storage products handle in different ways the synchronization or replication of SCSI reservations across storage arrays. If an engagement with Red Hat Support reveals that such a technology is in use and may be disrupting the behavior of either of these agents, Red Hat may require reproduction of the concern without that storage solution in use, or may require further investigation by the vendor into those conditions to determine if they are compatible with the agents' expectations.

• In environments utilizing fence_scsi or fence_mpath against storage devices spanning multiple storage arrays, the expectation is that all SCSI persistent reservations and registration keys will be automatically propagated to all arrays such that nodes throughout the cluster see the same view.

• If a storage-array "split" in connectivity occurs, it is expected that the storage arrays will have some mechanism by which they can ensure that actions carried out during a split are handled sanely. The state of data on these arrays cannot become divergent, and the storage solution must be able to guarantee that. Nodes interacting with reservations/registrations across the separate arrays should not be able to carry out conflicting operations in parallel.

• If a support engagements reveals that the storage technologies in use may be producing behavior outside these expectations, Red Hat Support may require removal of that technology or further investigation by the vendor in order to continue providing assistance.

Conditions with VMware platforms

fence_scsi and fence_mpath are subject to limitations in the storage-access method used on VMware platforms:

[1]: Use of RDM devices in Physical Compatibility mode is supported by Red Hat, under the following conditions:

• RHEL 7+
• Each VM in the cluster will always run on separate ESXi hosts - if there is possibility of VMs running on the same host, these agents may not function properly, and Red Hat will not assist with concerns in such configurations.
• Each VM accesses shared storage devices through a dedicated virtual SCSI controller using Sharing mode: Physical
• Each virtual SCSI controller must be configured with scsiX.releaseResvOnPowerOff set to FALSE in each VM's advanced configuration settings.
• For instruction in configuring these conditions, see: Administrative procedures - Configuring VMware VMs to be Compatible with the fence_scsi or fence_mpath STONITH Method

[2]: iSCSI devices must be accessed directly by VMs, with no passthrough of VMware layers (ex. VMs using shared storage presented via Virtual Disk Files (VMDKs)).

[3]: Currently fence_scsi and fence_mapth are unsupported on VMware VMDK with multi-writer option enabled and VMware "Clustered VMDK" storage devices.

See also Red Hat's full set of policies regarding High Availability on VMware or other platforms: Support policies for RHEL High Availability clusters

[4] VMware vSphere 7, VMware added support for SCSI-3 Persistent Reservations (SCSI-3 PR) at the virtual disk (VMDK) level BUT VMware does not support this on RHEL (and is not supported by RHEL). For more information see: [RHEL-7621] RFE: Add support of fence-scsi for RHEL 8 cluster running in vCenter 7/8 with VMDK shared disk