Chapter 7. Diagnosing and correcting problems with GFS2 file systems

This section provides information about some common GFS2 issues and how to address them.

7.1. GFS2 filesystem unavailable to a node (the GFS2 withdraw function)

The GFS2 withdraw function is a data integrity feature of the GFS2 file system that prevents potential file system damage due to faulty hardware or kernel software. If the GFS2 kernel module detects an inconsistency while using a GFS2 file system on any given cluster node, it withdraws from the file system, leaving it unavailable to that node until it is unmounted and remounted (or the machine detecting the problem is rebooted). All other mounted GFS2 file systems remain fully functional on that node. (The GFS2 withdraw function is less severe than a kernel panic, which causes the node to be fenced.)

The main categories of inconsistency that can cause a GFS2 withdraw are as follows:

  • Inode consistency error
  • Resource group consistency error
  • Journal consistency error
  • Magic number metadata consistency error
  • Metadata type consistency error

An example of an inconsistency that would cause a GFS2 withdraw is an incorrect block count for a file’s inode. When GFS2 deletes a file, it systematically removes all the data and metadata blocks referenced by that file. When done, it checks the inode’s block count. If the block count is not 1 (meaning all that is left is the disk inode itself), that indicates a file system inconsistency, since the inode’s block count did not match the actual blocks used for the file.

In many cases, the problem may have been caused by faulty hardware (faulty memory, motherboard, HBA, disk drives, cables, and so forth). It may also have been caused by a kernel bug (another kernel module accidentally overwriting GFS2’s memory), or actual file system damage (caused by a GFS2 bug).

In most cases, the best way to recover from a withdrawn GFS2 file system is to reboot or fence the node. The withdrawn GFS2 file system will give you an opportunity to relocate services to another node in the cluster. After services are relocated you can reboot the node or force a fence with this command.

# pcs stonith fence node
Warning

Do not try to unmount and remount the file system manually with the umount and mount commands. You must use the pcs command, otherwise Pacemaker will detect the file system service has disappeared and fence the node.

The consistency problem that caused the withdraw may make stopping the file system service impossible as it may cause the system to hang.

If the problem persists after a remount, you should stop the file system service to unmount the file system from all nodes in the cluster, then perform a file system check with the fsck.gfs2 command before restarting the service with the following procedure.

  1. Reboot the affected node.
  2. Disable the non-clone file system service in Pacemaker to unmount the file system from every node in the cluster.

    # pcs resource disable --wait=100 mydata_fs
  3. From one node of the cluster, run the fsck.gfs2 command on the file system device to check for and repair any file system damage.

    # fsck.gfs2 -y /dev/vg_mydata/mydata > /tmp/fsck.out
  4. Remount the GFS2 file system from all nodes by re-enabling the file system service:

    # pcs resource enable --wait=100 mydata_fs

You can override the GFS2 withdraw function by mounting the file system with the -o errors=panic option specified in the file system service.

# pcs resource update mydata_fs “options=noatime,errors=panic”

When this option is specified, any errors that would normally cause the system to withdraw force a kernel panic instead. This stops the node’s communications, which causes the node to be fenced. This is especially useful for clusters that are left unattended for long periods of time without monitoring or intervention.

Internally, the GFS2 withdraw function works by disconnecting the locking protocol to ensure that all further file system operations result in I/O errors. As a result, when the withdraw occurs, it is normal to see a number of I/O errors from the device mapper device reported in the system logs.

7.2. GFS2 file system hangs and requires reboot of one node

If your GFS2 file system hangs and does not return commands run against it, but rebooting one specific node returns the system to normal, this may be indicative of a locking problem or bug. Should this occur, gather GFS2 data during one of these occurences and open a support ticket with Red Hat Support, as described in Gathering GFS2 data for troubleshooting.

7.3. GFS2 file system hangs and requires reboot of all nodes

If your GFS2 file system hangs and does not return commands run against it, requiring that you reboot all nodes in the cluster before using it, check for the following issues.

  • You may have had a failed fence. GFS2 file systems will freeze to ensure data integrity in the event of a failed fence. Check the messages logs to see if there are any failed fences at the time of the hang. Ensure that fencing is configured correctly.
  • The GFS2 file system may have withdrawn. Check through the messages logs for the word withdraw and check for any messages and call traces from GFS2 indicating that the file system has been withdrawn. A withdraw is indicative of file system corruption, a storage failure, or a bug. At the earliest time when it is convenient to unmount the file system, you should perform the following procedure:

    1. Reboot the node on which the withdraw occurred.

      # /sbin/reboot
    2. Stop the file system resource to unmount the GFS2 file system on all nodes.

      # pcs resource disable --wait=100 mydata_fs
    3. Capture the metadata with the gfs2_edit savemeta…​ command. You should ensure that there is sufficient space for the file, which in some cases may be large. In this example, the metadata is saved to a file in the /root directory.

      # gfs2_edit savemeta /dev/vg_mydata/mydata /root/gfs2metadata.gz
    4. Update the gfs2-utils package.

      # sudo yum update gfs2-utils
    5. On one node, run the fsck.gfs2 command on the file system to ensure file system integrity and repair any damage.

      # fsck.gfs2 -y /dev/vg_mydata/mydata > /tmp/fsck.out
    6. After the fsck.gfs2 command has completed, re-enable the file system resource to return it to service:

      # pcs resource enable --wait=100 mydata_fs
    7. Open a support ticket with Red Hat Support. Inform them you experienced a GFS2 withdraw and provide logs and the debugging information generated by the sosreports and gfs2_edit savemeta commands.

      In some instances of a GFS2 withdraw, commands can hang that are trying to access the file system or its block device. In these cases a hard reboot is required to reboot the cluster.

      For information on the GFS2 withdraw function, see GFS2 filesystem unavailable to a node (the GFS2 withdraw function).

  • This error may be indicative of a locking problem or bug. Gather data during one of these occurrences and open a support ticket with Red Hat Support, as described in Gathering GFS2 data for troubleshooting.

7.4. GFS2 file system does not mount on newly added cluster node

If you add a new node to a cluster and find that you cannot mount your GFS2 file system on that node, you may have fewer journals on the GFS2 file system than nodes attempting to access the GFS2 file system. You must have one journal per GFS2 host you intend to mount the file system on (with the exception of GFS2 file systems mounted with the spectator mount option set, since these do not require a journal). You can add journals to a GFS2 file system with the gfs2_jadd command. Adding journals to a GFS2 file system.

7.5. Space indicated as used in empty file system

If you have an empty GFS2 file system, the df command will show that there is space being taken up. This is because GFS2 file system journals consume space (number of journals * journal size) on disk. If you created a GFS2 file system with a large number of journals or specified a large journal size then you will be see (number of journals * journal size) as already in use when you execute the df command. Even if you did not specify a large number of journals or large journals, small GFS2 file systems (in the 1GB or less range) will show a large amount of space as being in use with the default GFS2 journal size.

7.6. Gathering GFS2 data for troubleshooting

If your GFS2 file system hangs and does not return commands run against it and you find that you need to open a ticket with Red Hat Support, you should first gather the following data:

  • The GFS2 lock dump for the file system on each node:

    cat /sys/kernel/debug/gfs2/fsname/glocks >glocks.fsname.nodename
  • The DLM lock dump for the file system on each node: You can get this information with the dlm_tool:

    dlm_tool lockdebug -sv lsname.

    In this command, lsname is the lockspace name used by DLM for the file system in question. You can find this value in the output from the group_tool command.

  • The output from the sysrq -t command.
  • The contents of the /var/log/messages file.

Once you have gathered that data, you can open a ticket with Red Hat Support and provide the data you have collected.