Chapter 6. Configuring an active/passive NFS server in a Red Hat High Availability cluster
The following procedure configures a highly available active/passive NFS server on a two-node Red Hat Enterprise Linux High Availability Add-On cluster using shared storage. The procedure uses pcs
to configure Pacemaker cluster resources. In this use case, clients access the NFS file system through a floating IP address. The NFS server runs on one of two nodes in the cluster. If the node on which the NFS server is running becomes inoperative, the NFS server starts up again on the second node of the cluster with minimal service interruption.
6.1. Prerequisites
This use case requires that your system include the following components:
- A two-node Red Hat High Availability cluster with power fencing configured for each node. We recommend but do not require a private network. This procedure uses the cluster example provided in Creating a Red Hat High-Availability cluster with Pacemaker.
- A public virtual IP address, required for the NFS server.
- Shared storage for the nodes in the cluster, using iSCSI, Fibre Channel, or other shared network block device.
6.2. Procedural overview
Configuring a highly available active/passive NFS server on an existing two-node Red Hat Enterprise Linux High Availability cluster requires that you perform the following steps:
-
Configure an
ext4
file system on the LVM logical volumemy_lv
on the shared storage for the nodes in the cluster. - Configure an NFS share on the shared storage on the LVM logical volume.
- Create the cluster resources.
- Test the NFS server you have configured.
6.3. Configuring an LVM volume with an ext4 file system in a Pacemaker cluster
This use case requires that you create an LVM logical volume on storage that is shared between the nodes of the cluster.
LVM volumes and the corresponding partitions and devices used by cluster nodes must be connected to the cluster nodes only.
The following procedure creates an LVM logical volume and then creates an ext4 file system on that volume for use in a Pacemaker cluster. In this example, the shared partition /dev/sdb1
is used to store the LVM physical volume from which the LVM logical volume will be created.
On both nodes of the cluster, perform the following steps to set the value for the LVM system ID to the value of the
uname
identifier for the system. The LVM system ID will be used to ensure that only the cluster is capable of activating the volume group.Set the
system_id_source
configuration option in the/etc/lvm/lvm.conf
configuration file touname
.# Configuration option global/system_id_source. system_id_source = "uname"
Verify that the LVM system ID on the node matches the
uname
for the node.#
lvm systemid
system ID: z1.example.com #uname -n
z1.example.com
Create the LVM volume and create an ext4 file system on that volume. Since the
/dev/sdb1
partition is storage that is shared, you perform this part of the procedure on one node only.Create an LVM physical volume on partition
/dev/sdb1
.#
pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully createdCreate the volume group
my_vg
that consists of the physical volume/dev/sdb1
.#
vgcreate my_vg /dev/sdb1
Volume group "my_vg" successfully createdVerify that the new volume group has the system ID of the node on which you are running and from which you created the volume group.
#
vgs -o+systemid
VG #PV #LV #SN Attr VSize VFree System ID my_vg 1 0 0 wz--n- <1.82t <1.82t z1.example.comCreate a logical volume using the volume group
my_vg
.#
lvcreate -L450 -n my_lv my_vg
Rounding up size to full physical extent 452.00 MiB Logical volume "my_lv" createdYou can use the
lvs
command to display the logical volume.#
lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert my_lv my_vg -wi-a---- 452.00m ...Create an ext4 file system on the logical volume
my_lv
.#
mkfs.ext4 /dev/my_vg/my_lv
mke2fs 1.44.3 (10-July-2018) Creating filesystem with 462848 1k blocks and 115824 inodes ...
6.5. Configuring the resources and resource group for an NFS server in a cluster
This section provides the procedure for configuring the cluster resources for this use case.
If you have not configured a fencing device for your cluster, by default the resources do not start.
If you find that the resources you configured are not running, you can run the pcs resource debug-start resource
command to test the resource configuration. This starts the service outside of the cluster’s control and knowledge. At the point the configured resources are running again, run pcs resource cleanup resource
to make the cluster aware of the updates.
The following procedure configures the system resources. To ensure these resources all run on the same node, they are configured as part of the resource group nfsgroup
. The resources will start in the order in which you add them to the group, and they will stop in the reverse order in which they are added to the group. Run this procedure from one node of the cluster only.
Create the LVM-activate resource named
my_lvm
. Because the resource groupnfsgroup
does not yet exist, this command creates the resource group.WarningDo not configure more than one
LVM-activate
resource that uses the same LVM volume group in an active/passive HA configuration, as this risks data corruption. Additionally, do not configure anLVM-activate
resource as a clone resource in an active/passive HA configuration.[root@z1 ~]#
pcs resource create my_lvm ocf:heartbeat:LVM-activate
vgname=my_vg
vg_access_mode=system_id --group nfsgroup
Check the status of the cluster to verify that the resource is running.
root@z1 ~]#
pcs status
Cluster name: my_cluster Last updated: Thu Jan 8 11:13:17 2015 Last change: Thu Jan 8 11:13:08 2015 Stack: corosync Current DC: z2.example.com (2) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 3 Resources configured Online: [ z1.example.com z2.example.com ] Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM): Started z1.example.com PCSD Status: z1.example.com: Online z2.example.com: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabledConfigure a
Filesystem
resource for the cluster.The following command configures an ext4
Filesystem
resource namednfsshare
as part of thenfsgroup
resource group. This file system uses the LVM volume group and ext4 file system you created in Configuring an LVM volume with an ext4 file system and will be mounted on the/nfsshare
directory you created in Configuring an NFS share.[root@z1 ~]#
pcs resource create nfsshare Filesystem
\device=/dev/my_vg/my_lv directory=/nfsshare
\fstype=ext4 --group nfsgroup
You can specify mount options as part of the resource configuration for a
Filesystem
resource with theoptions=options
parameter. Run thepcs resource describe Filesystem
command for full configuration options.Verify that the
my_lvm
andnfsshare
resources are running.[root@z1 ~]#
pcs status
... Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com ...Create the
nfsserver
resource namednfs-daemon
as part of the resource groupnfsgroup
.NoteThe
nfsserver
resource allows you to specify annfs_shared_infodir
parameter, which is a directory that NFS servers use to store NFS-related stateful information.It is recommended that this attribute be set to a subdirectory of one of the
Filesystem
resources you created in this collection of exports. This ensures that the NFS servers are storing their stateful information on a device that will become available to another node if this resource group needs to relocate. In this example;-
/nfsshare
is the shared-storage directory managed by theFilesystem
resource -
/nfsshare/exports/export1
and/nfsshare/exports/export2
are the export directories -
/nfsshare/nfsinfo
is the shared-information directory for thenfsserver
resource
[root@z1 ~]#
pcs resource create nfs-daemon nfsserver
\nfs_shared_infodir=/nfsshare/nfsinfo nfs_no_notify=true
\--group nfsgroup
[root@z1 ~]#pcs status
...-
Add the
exportfs
resources to export the/nfsshare/exports
directory. These resources are part of the resource groupnfsgroup
. This builds a virtual directory for NFSv4 clients. NFSv3 clients can access these exports as well.NoteThe
fsid=0
option is required only if you want to create a virtual directory for NFSv4 clients. For more information, see How do I configure the fsid option in an NFS server’s /etc/exports file?.[root@z1 ~]#
pcs resource create nfs-root exportfs
\clientspec=192.168.122.0/255.255.255.0
\options=rw,sync,no_root_squash
\directory=/nfsshare/exports
\fsid=0 --group nfsgroup
[root@z1 ~]# #pcs resource create nfs-export1 exportfs
\clientspec=192.168.122.0/255.255.255.0
\options=rw,sync,no_root_squash directory=/nfsshare/exports/export1
\fsid=1 --group nfsgroup
[root@z1 ~]# #pcs resource create nfs-export2 exportfs
\clientspec=192.168.122.0/255.255.255.0
\options=rw,sync,no_root_squash directory=/nfsshare/exports/export2
\fsid=2 --group nfsgroup
Add the floating IP address resource that NFS clients will use to access the NFS share. This resource is part of the resource group
nfsgroup
. For this example deployment, we are using 192.168.122.200 as the floating IP address.[root@z1 ~]#
pcs resource create nfs_ip IPaddr2
\ip=192.168.122.200 cidr_netmask=24 --group nfsgroup
Add an
nfsnotify
resource for sending NFSv3 reboot notifications once the entire NFS deployment has initialized. This resource is part of the resource groupnfsgroup
.NoteFor the NFS notification to be processed correctly, the floating IP address must have a host name associated with it that is consistent on both the NFS servers and the NFS client.
[root@z1 ~]#
pcs resource create nfs-notify nfsnotify
\source_host=192.168.122.200 --group nfsgroup
After creating the resources and the resource constraints, you can check the status of the cluster. Note that all resources are running on the same node.
[root@z1 ~]#
pcs status
... Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z1.example.com nfs-root (ocf::heartbeat:exportfs): Started z1.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z1.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z1.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z1.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z1.example.com ...
6.6. Testing the NFS resource configuration
You can validate your system configuration with the following procedures. You should be able to mount the exported file system with either NFSv3 or NFSv4.
6.6.1. Testing the NFS export
On a node outside of the cluster, residing in the same network as the deployment, verify that the NFS share can be seen by mounting the NFS share. For this example, we are using the 192.168.122.0/24 network.
#
showmount -e 192.168.122.200
Export list for 192.168.122.200: /nfsshare/exports/export1 192.168.122.0/255.255.255.0 /nfsshare/exports 192.168.122.0/255.255.255.0 /nfsshare/exports/export2 192.168.122.0/255.255.255.0To verify that you can mount the NFS share with NFSv4, mount the NFS share to a directory on the client node. After mounting, verify that the contents of the export directories are visible. Unmount the share after testing.
#
mkdir nfsshare
#mount -o "vers=4" 192.168.122.200:export1 nfsshare
#ls nfsshare
clientdatafile1 #umount nfsshare
Verify that you can mount the NFS share with NFSv3. After mounting, verify that the test file
clientdatafile1
is visible. Unlike NFSv4, since NFSv3 does not use the virtual file system, you must mount a specific export. Unmount the share after testing.#
mkdir nfsshare
#mount -o "vers=3" 192.168.122.200:/nfsshare/exports/export2 nfsshare
#ls nfsshare
clientdatafile2 #umount nfsshare
6.6.2. Testing for failover
On a node outside of the cluster, mount the NFS share and verify access to the
clientdatafile1
we created in Configuring an NFS share#
mkdir nfsshare
#mount -o "vers=4" 192.168.122.200:export1 nfsshare
#ls nfsshare
clientdatafile1From a node within the cluster, determine which node in the cluster is running
nfsgroup
. In this example,nfsgroup
is running onz1.example.com
.[root@z1 ~]#
pcs status
... Full list of resources: myapc (stonith:fence_apc_snmp): Started z1.example.com Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM): Started z1.example.com nfsshare (ocf::heartbeat:Filesystem): Started z1.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z1.example.com nfs-root (ocf::heartbeat:exportfs): Started z1.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z1.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z1.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z1.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z1.example.com ...From a node within the cluster, put the node that is running
nfsgroup
in standby mode.[root@z1 ~]#
pcs node standby z1.example.com
Verify that
nfsgroup
successfully starts on the other cluster node.[root@z1 ~]#
pcs status
... Full list of resources: Resource Group: nfsgroup my_lvm (ocf::heartbeat:LVM): Started z2.example.com nfsshare (ocf::heartbeat:Filesystem): Started z2.example.com nfs-daemon (ocf::heartbeat:nfsserver): Started z2.example.com nfs-root (ocf::heartbeat:exportfs): Started z2.example.com nfs-export1 (ocf::heartbeat:exportfs): Started z2.example.com nfs-export2 (ocf::heartbeat:exportfs): Started z2.example.com nfs_ip (ocf::heartbeat:IPaddr2): Started z2.example.com nfs-notify (ocf::heartbeat:nfsnotify): Started z2.example.com ...From the node outside the cluster on which you have mounted the NFS share, verify that this outside node still continues to have access to the test file within the NFS mount.
#
ls nfsshare
clientdatafile1Service will be lost briefly for the client during the failover but the client should recover it with no user intervention. By default, clients using NFSv4 may take up to 90 seconds to recover the mount; this 90 seconds represents the NFSv4 file lease grace period observed by the server on startup. NFSv3 clients should recover access to the mount in a matter of a few seconds.
From a node within the cluster, remove the node that was initially running running
nfsgroup
from standby mode.NoteRemoving a node from
standby
mode does not in itself cause the resources to fail back over to that node. This will depend on theresource-stickiness
value for the resources. For information on theresource-stickiness
meta attribute, see Configuring a resource to prefer its current node.[root@z1 ~]#
pcs node unstandby z1.example.com