Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 3. Administering GFS2 file systems

There are a variety of commands and options that you use to create, mount, grow, and manage GFS2 file systems.

3.1. GFS2 file system creation

You create a GFS2 file system with the mkfs.gfs2 command. A file system is created on an activated LVM volume.

3.1.1. The GFS2 mkfs command

The following information is required to run the mkfs.gfs2 command to create a clustered GFS2 file system:

  • Lock protocol/module name, which is lock_dlm for a cluster
  • Cluster name
  • Number of journals (one journal required for each node that may be mounting the file system)
Note

Once you have created a GFS2 file system with the mkfs.gfs2 command, you cannot decrease the size of the file system. You can, however, increase the size of an existing file system with the gfs2_grow command.

The format for creating a clustered GFS2 file system is as follows. Note that Red Hat does not support the use of GFS2 as a single-node file system. 

mkfs.gfs2 -p lock_dlm -t ClusterName:FSName -j NumberJournals BlockDevice

If you prefer, you can create a GFS2 file system by using the mkfs command with the -t parameter specifying a file system of type gfs2, followed by the GFS2 file system options. 

mkfs -t gfs2 -p lock_dlm -t ClusterName:FSName -j NumberJournals BlockDevice
Warning

Improperly specifying the ClusterName:FSName parameter may cause file system or lock space corruption.

ClusterName
The name of the cluster for which the GFS2 file system is being created.
FSName
The file system name, which can be 1 to 16 characters long. The name must be unique for all lock_dlm file systems over the cluster.
NumberJournals
Specifies the number of journals to be created by the mkfs.gfs2 command. One journal is required for each node that mounts the file system. For GFS2 file systems, more journals can be added later without growing the file system.
BlockDevice
Specifies a logical or other block device

The following table describes the mkfs.gfs2 command options (flags and parameters).

Table 3.1. Command Options: mkfs.gfs2

FlagParameterDescription

-c

Megabytes

Sets the initial size of each journal’s quota change file to Megabytes.

-D

 

Enables debugging output.

-h

 

Help. Displays available options.

-J

Megabytes

Specifies the size of the journal in megabytes. Default journal size is 128 megabytes. The minimum size is 8 megabytes. Larger journals improve performance, although they use more memory than smaller journals.

-j

Number

Specifies the number of journals to be created by the mkfs.gfs2 command. One journal is required for each node that mounts the file system. If this option is not specified, one journal will be created. For GFS2 file systems, you can add additional journals at a later time without growing the file system.

-O

 

Prevents the mkfs.gfs2 command from asking for confirmation before writing the file system.

-p

LockProtoName

* Specifies the name of the locking protocol to use. Recognized locking protocols include:

* lock_dlm — The standard locking module, required for a clustered file system.

* lock_nolock — Used when GFS2 is acting as a local file system (one node only). Red Hat does not support the use of GFS2 as a single-node file system in a production environment. lock_nolock should be used only for the purposes of backup or for a secondary-site Disaster Recovery node, as described in Minimum cluster size. When using lock_nolock, you must ensure that the GFS2 file system is being used by only one system at a time.

-q

 

Quiet. Do not display anything.

-r

Megabytes

Specifies the size of the resource groups in megabytes. The minimum resource group size is 32 megabytes. The maximum resource group size is 2048 megabytes. A large resource group size may increase performance on very large file systems. If this is not specified, mkfs.gfs2 chooses the resource group size based on the size of the file system: average size file systems will have 256 megabyte resource groups, and bigger file systems will have bigger resource groups for better performance.

-t

LockTableName

* A unique identifier that specifies the lock table field when you use the lock_dlm protocol; the lock_nolock protocol does not use this parameter.

* This parameter has two parts separated by a colon (no spaces) as follows: ClusterName:FSName.

* ClusterName is the name of the cluster for which the GFS2 file system is being created; only members of this cluster are permitted to use this file system.

* FSName, the file system name, can be 1 to 16 characters in length, and the name must be unique among all file systems in the cluster.

-V

 

Displays command version information.

3.1.2. Creating a GFS2 file system

The following example creates two GFS2 file systems. For both of these file systems, lock_dlm` is the locking protocol that the file system uses, since this is a clustered file system. Both file systems can be used in the cluster named alpha.

For the first file system, file system name is mydata1. it contains eight journals and is created on /dev/vg01/lvol0. For the second file system, the file system name is mydata2. It contains eight journals and is created on /dev/vg01/lvol1. 

# mkfs.gfs2 -p lock_dlm -t alpha:mydata1 -j 8 /dev/vg01/lvol0
# mkfs.gfs2 -p lock_dlm -t alpha:mydata2 -j 8 /dev/vg01/lvol1

3.2. Mounting a GFS2 file system

Before you can mount a GFS2 file system, the file system must exist, the volume where the file system exists must be activated, and the supporting clustering and locking systems must be started. After those requirements have been met, you can mount the GFS2 file system as you would any Linux file system.

Note

You should always use Pacemaker to manage the GFS2 file system in a production environment rather than manually mounting the file system with a mount command, as this may cause issues at system shutdown.

To manipulate file ACLs, you must mount the file system with the -o acl mount option. If a file system is mounted without the -o acl mount option, users are allowed to view ACLs (with getfacl), but are not allowed to set them (with setfacl).

3.2.1. Mounting a GFS2 file system with no options specified

In this example, the GFS2 file system on /dev/vg01/lvol0 is mounted on the /mygfs2 directory. 

# mount /dev/vg01/lvol0 /mygfs2

3.2.2. Mounting a GFS2 file system that specifies mount options

The following is the format for the command to mount a GFS2 file system that specifies mount options. 

mount BlockDevice MountPoint -o option
BlockDevice
Specifies the block device where the GFS2 file system resides.
MountPoint
Specifies the directory where the GFS2 file system should be mounted.

The -o option argument consists of GFS2-specific options or acceptable standard Linux mount -o options, or a combination of both. Multiple option parameters are separated by a comma and no spaces.

Note

The mount command is a Linux system command. In addition to using these GFS2-specific options, you can use other, standard, mount command options (for example, -r). For information about other Linux mount command options, see the Linux mount man page.

The following table describes the available GFS2-specific -o option values that can be passed to GFS2 at mount time.

Note

This table includes descriptions of options that are used with local file systems only. Note, however, that Red Hat does not support the use of GFS2 as a single-node file system. Red Hat will continue to support single-node GFS2 file systems for mounting snapshots of cluster file systems (for example, for backup purposes).

Table 3.2. GFS2-Specific Mount Options

OptionDescription

acl

Allows manipulating file ACLs. If a file system is mounted without the acl mount option, users are allowed to view ACLs (with getfacl), but are not allowed to set them (with setfacl).

data=[ordered|writeback]

When data=ordered is set, the user data modified by a transaction is flushed to the disk before the transaction is committed to disk. This should prevent the user from seeing uninitialized blocks in a file after a crash. When data=writeback mode is set, the user data is written to the disk at any time after it is dirtied; this does not provide the same consistency guarantee as ordered mode, but it should be slightly faster for some workloads. The default value is ordered mode.

ignore_local_fs

Caution: This option should not be used when GFS2 file systems are shared.

Forces GFS2 to treat the file system as a multi-host file system. By default, using lock_nolock automatically turns on the localflocks flag.

localflocks

Caution: This option should not be used when GFS2 file systems are shared.

Tells GFS2 to let the VFS (virtual file system) layer do all flock and fcntl. The localflocks flag is automatically turned on by lock_nolock.

lockproto=LockModuleName

Allows the user to specify which locking protocol to use with the file system. If LockModuleName is not specified, the locking protocol name is read from the file system superblock.

locktable=LockTableName

Allows the user to specify which locking table to use with the file system.

quota=[off/account/on]

Turns quotas on or off for a file system. Setting the quotas to be in the account state causes the per UID/GID usage statistics to be correctly maintained by the file system; limit and warn values are ignored. The default value is off.

errors=panic|withdraw

When errors=panic is specified, file system errors will cause a kernel panic. When errors=withdraw is specified, which is the default behavior, file system errors will cause the system to withdraw from the file system and make it inaccessible until the next reboot; in some cases the system may remain running.

discard/nodiscard

Causes GFS2 to generate "discard" I/O requests for blocks that have been freed. These can be used by suitable hardware to implement thin provisioning and similar schemes.

barrier/nobarrier

Causes GFS2 to send I/O barriers when flushing the journal. The default value is on. This option is automatically turned off if the underlying device does not support I/O barriers. Use of I/O barriers with GFS2 is highly recommended at all times unless the block device is designed so that it cannot lose its write cache content (for example, if it is on a UPS or it does not have a write cache).

quota_quantum=secs

Sets the number of seconds for which a change in the quota information may sit on one node before being written to the quota file. This is the preferred way to set this parameter. The value is an integer number of seconds greater than zero. The default is 60 seconds. Shorter settings result in faster updates of the lazy quota information and less likelihood of someone exceeding their quota. Longer settings make file system operations involving quotas faster and more efficient.

statfs_quantum=secs

Setting statfs_quantum to 0 is the preferred way to set the slow version of statfs. The default value is 30 secs which sets the maximum time period before statfs changes will be synced to the master statfs file. This can be adjusted to allow for faster, less accurate statfs values or slower more accurate values. When this option is set to 0, statfs will always report the true values.

statfs_percent=value

Provides a bound on the maximum percentage change in the statfs information about a local basis before it is synced back to the master statfs file, even if the time period has not expired. If the setting of statfs_quantum is 0, then this setting is ignored.

3.2.3. Unmounting a GFS2 file system

GFS2 file systems that have been mounted manually rather than automatically through Pacemaker will not be known to the system when file systems are unmounted at system shutdown. As a result, the GFS2 resource agent will not unmount the GFS2 file system. After the GFS2 resource agent is shut down, the standard shutdown process kills off all remaining user processes, including the cluster infrastructure, and tries to unmount the file system. This unmount will fail without the cluster infrastructure and the system will hang.

To prevent the system from hanging when the GFS2 file systems are unmounted, you should do one of the following:

  • Always use Pacemaker to manage the GFS2 file system.
  • If a GFS2 file system has been mounted manually with the mount command, be sure to unmount the file system manually with the umount command before rebooting or shutting down the system.

If your file system hangs while it is being unmounted during system shutdown under these circumstances, perform a hardware reboot. It is unlikely that any data will be lost since the file system is synced earlier in the shutdown process.

The GFS2 file system can be unmounted the same way as any Linux file system, by using the umount command.

Note

The umount command is a Linux system command. Information about this command can be found in the Linux umount command man pages.

Usage 

umount MountPoint
MountPoint
Specifies the directory where the GFS2 file system is currently mounted.

3.3. Backing up a GFS2 file system

It is important to make regular backups of your GFS2 file system in case of emergency, regardless of the size of your file system. Many system administrators feel safe because they are protected by RAID, multipath, mirroring, snapshots, and other forms of redundancy, but there is no such thing as safe enough.

It can be a problem to create a backup since the process of backing up a node or set of nodes usually involves reading the entire file system in sequence. If this is done from a single node, that node will retain all the information in cache until other nodes in the cluster start requesting locks. Running this type of backup program while the cluster is in operation will negatively impact performance.

Dropping the caches once the backup is complete reduces the time required by other nodes to regain ownership of their cluster locks and caches. This is still not ideal, however, because the other nodes will have stopped caching the data that they were caching before the backup process began. You can drop caches using the following command after the backup is complete: 

echo -n 3 > /proc/sys/vm/drop_caches

It is faster if each node in the cluster backs up its own files so that the task is split between the nodes. You might be able to accomplish this with a script that uses the rsync command on node-specific directories.

Red Hat recommends making a GFS2 backup by creating a hardware snapshot on the SAN, presenting the snapshot to another system, and backing it up there. The backup system should mount the snapshot with -o lockproto=lock_nolock since it will not be in a cluster. Note, however, that Red Hat does not support the use of GFS2 as a single-node file system in a production environment. This option should be used only for the purposes of backup or for a secondary-site Disaster Recovery node, as described in Minimum cluster size. When using this option, you must ensure that the GFS2 file system is being used by only one system at a time.

3.4. Suspending activity on a GFS2 file system

You can suspend write activity to a file system by using the dmsetup suspend command. Suspending write activity allows hardware-based device snapshots to be used to capture the file system in a consistent state. The dmsetup resume command ends the suspension.

The format for the command to suspend activity on a GFS2 file system is as follows. 

dmsetup suspend MountPoint

This example suspends writes to file system /mygfs2. 

# dmsetup suspend /mygfs2

The format for the command to end suspension of activity on a GFS2 file system is as follows. 

dmsetup resume MountPoint

This example ends suspension of writes to file system /mygfs2. 

# dmsetup resume /mygfs2

3.5. Growing a GFS2 file system

The gfs2_grow command is used to expand a GFS2 file system after the device where the file system resides has been expanded. Running the gfs2_grow command on an existing GFS2 file system fills all spare space between the current end of the file system and the end of the device with a newly initialized GFS2 file system extension. All nodes in the cluster can then use the extra storage space that has been added.

Note

You cannot decrease the size of a GFS2 file system.

The gfs2_grow command must be run on a mounted file system. The following procedure increases the size of the GFS2 file system in a cluster that is mounted on the logical volume shared_vg/shared_lv1 with a mount point of /mnt/gfs2.

Procedure

  1. Perform a backup of the data on the file system.

  2. If you do not know the logical volume that is used by the file system to be expanded, you can determine this by running the df mountpoint command. This will display the device name in the following format:

    /dev/mapper/vg-lv

    For example, the device name /dev/mapper/shared_vg-shared_lv1 indicates that the logical volume is shared_vg/shared_lv1.

  3. On one node of the cluster, expand the underlying cluster volume with the lvextend command. If you are running RHEL 8.0, use the --lockopt skiplv option to override normal logical volume locking. This is not necessary for systems running RHEL 8.1 or later.

    For RHEL 8.1 and later use the following command. 

    # lvextend -L+1G shared_vg/shared_lv1
Size of logical volume shared_vg/shared_lv1 changed from 5.00 GiB (1280 extents) to 6.00 GiB (1536 extents).
WARNING: extending LV with a shared lock, other hosts may require LV refresh.
Logical volume shared_vg/shared_lv1 successfully resized.

    For RHEL 8.0, use the following command. 

    # lvextend --lockopt skiplv -L+1G shared_vg/shared_lv1
WARNING: skipping LV lock in lvmlockd.
Size of logical volume shared_vg/shared_lv1 changed from 5.00 GiB (1280 extents) to 6.00 GiB (1536 extents).
WARNING: extending LV with a shared lock, other hosts may require LV refresh.
Logical volume shared_vg/shared_lv1 successfully resized.

  4. If you are running RHEL 8.0, on every additional node of the cluster refresh the logical volume to update the active logical volume on that node. This step is not necessary on systems running RHEL 8.1 and later as the step is automated when the logical volume is extended. 

    # lvchange --refresh shared_vg/shared_lv1

  5. One one node of the cluster, increase the size of the GFS2 file system. Do not extend the file system if the logical volume was not refreshed on all of the nodes, otherwise the file system data may become unavailable throughout the cluster. 

    # gfs2_grow /mnt/gfs2
FS: Mount point:             /mnt/gfs2
FS: Device:                  /dev/mapper/shared_vg-shared_lv1
FS: Size:                    1310719 (0x13ffff)
DEV: Length:                 1572864 (0x180000)
The file system will grow by 1024MB.
gfs2_grow complete.

  6. Run the df command on all nodes to check that the new space is now available in the file system. Note that it may take up to 30 seconds for the df command on all nodes to show the same file system size 

    # df -h /mnt/gfs2]
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/shared_vg-shared_lv1  6.0G  4.5G  1.6G  75% /mnt/gfs2

3.6. Adding journals to a GFS2 file system

GFS2 requires one journal for each node in a cluster that needs to mount the file system. If you add additional nodes to the cluster, you can add journals to a GFS2 file system with the gfs2_jadd command. You can add journals to a GFS2 file system dynamically at any point without expanding the underlying logical volume. The gfs2_jadd command must be run on a mounted file system, but it needs to be run on only one node in the cluster. All the other nodes sense that the expansion has occurred.

Note

If a GFS2 file system is full, the gfs2_jadd command will fail, even if the logical volume containing the file system has been extended and is larger than the file system. This is because in a GFS2 file system, journals are plain files rather than embedded metadata, so simply extending the underlying logical volume will not provide space for the journals.

Before adding journals to a GFS2 file system, you can find out how many journals the GFS2 file system currently contains with the gfs2_edit -p jindex command, as in the following example: 

# gfs2_edit -p jindex /dev/sasdrives/scratch|grep journal
   3/3 [fc7745eb] 4/25 (0x4/0x19): File    journal0
   4/4 [8b70757d] 5/32859 (0x5/0x805b): File    journal1
   5/5 [127924c7] 6/65701 (0x6/0x100a5): File    journal2

The format for the basic command to add journals to a GFS2 file system is as follows. 

gfs2_jadd -j Number MountPoint
Number
Specifies the number of new journals to be added.
MountPoint
Specifies the directory where the GFS2 file system is mounted.

In this example, one journal is added to the file system on the /mygfs2 directory. 

# gfs2_jadd -j 1 /mygfs2