How do I recover from a gfs2 withdrawal and fix filesystem corruption on a Red Hat Enterprise Linux High Availability cluster?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux Server 7, 8, 9 (with the High Availability Add On and Resilient Storage Add Ons)
  • A Global Filesystem 2(gfs2)

Issue

  • We can see a withdrawal message in the /var/log/messages. Please let us know how can we recover the filesystem?
  • My gfs2 file system is reporting invalid metadata blocks. What do I do to fix this and prevent it from occurring again?
  • A node keeps panicking with gfs2_meta_indirect_buffer in the backtrace when I have errors=panic in the mount options for a gfs2 file system.
  • A withdrawal happened on a gfs2 file system. Could this have left corruption in the metadata that needs to be fixed?
  • A withdrawal happened on a gfs2 file system. Could this have been caused by corruption?
Jun 19 00:22:34 node1 kernel: gfs2: fsid=my_cluster:fs1.1: fatal: invalid metadata block#012 bh = 1506986631 (type: exp=10, found=4)#012 function = ea_foreach_i, file = fs/gfs2/xattr.c, line = 86
Jun 19 00:22:34 node1 kernel: gfs2: fsid=my_cluster:fs1.1: about to withdraw this file system
Jun 19 00:22:34 node1 kernel: gfs2: fsid=my_cluster:fs1.1: telling LM to unmount
Jun 19 00:22:34 node1 kernel: dlm: fs1: leaving the lockspace group...
Jun 19 00:22:34 node1 kernel: dlm: fs1: group event done 0 0
Jun 19 00:22:34 node1 kernel: dlm: fs1: release_lockspace final free
Jun 19 00:22:34 node1 kernel: gfs2: fsid=my_cluster:fs1.1: withdrawn

Resolution

A gfs2 withdrawal prevents access to the filesystem and requires a reboot of the affected node before the LVM logical volume can be accessed again. After rebooting each node that had the withdrawal, run fsck.gfs2 (recommended) to find and fix any filesystem errors. Optionally, capture the filesystem metadata first so the cause of the withdrawal can be diagnosed later.

Important

  • Run fsck.gfs2 against the LVM logical volume that contains the gfs2 filesystem. Specifying the wrong device, such as an underlying physical volume (PV), can corrupt the on-disk LVM2 metadata. See The tool fsck.gfs2 was run against the wrong device and destroyed the LVM metadata for more information.
  • The filesystem must be unmounted on all cluster nodes before running fsck.gfs2 or capturing metadata. Running either against a filesystem that is still mounted on any node can cause corruption or produce invalid results.
  • Red Hat recommends running fsck.gfs2 to completion rather than interrupting it once repairs are underway.
  • Pressing Ctrl+C while fsck.gfs2 is running does not immediately stop it. It interrupts processing and displays a prompt asking whether to abort the command, skip the rest of the current pass, or continue processing (see Repairing a GFS2 File System, under "Note"). To cancel the check, choose abort at that prompt.
  • If there is a limited outage window and fsck.gfs2 cannot complete in time, reverting to a backup or a SAN snapshot is a workable alternative to waiting for the check to finish or aborting partway through.

Note

  • Capturing metadata and running fsck.gfs2 can take a long time, depending on the size and state of the filesystem. See How long does it take to fsck.gfs2 or capture the metadata for a gfs2 file system on RHEL?
  • If the node must return to production quickly, both steps can be skipped, at the cost of losing the information needed to diagnose the cause.
  • An fsck.gfs2 is required if filesystem errors prevent the filesystem from mounting, and is recommended at the earliest maintenance window otherwise.

Filesystem Check Procedure

  1. Reboot every node on which the filesystem withdrew. The filesystem cannot be mounted or accessed on a node, to capture metadata or run fsck.gfs2, until that node is rebooted. See the Diagnostic steps to find which nodes withdrew.

  2. Unmount the filesystem on all cluster nodes. If pacemaker manages the filesystem, disable the cloned Filesystem resource, which unmounts it on every node. On RHEL 8 or later, leave the LVM-activate resource enabled so the logical volume stays active.

    # pcs resource disable <filesystem_resource>

    Verify on each node that the filesystem is no longer mounted before continuing.

    # mount | grep gfs2

  3. (Optional) Capture the filesystem metadata before running fsck.gfs2. Checking the filesystem can fix the issue that caused the withdrawal, or change the state of the filesystem so that the cause can no longer be determined. For the procedure, see How do I save GFS2 metadata for troubleshooting on Red Hat Enterprise Linux 7, 8, and 9?

  4. Update the gfs2-utils package on the node that will run the check. The latest fsck.gfs2 has the latest optimizations and can fix the most known issues.

    # yum update gfs2-utils

  5. From one node only, run fsck.gfs2 against the logical volume and save the output to a file for review. The -y option answers "yes" to all questions, so metadata may be changed during the run. Omit -y to be prompted for each change, or use -n to make no changes.

    # fsck.gfs2 -v -y <device> 2>&1 | tee <output_file>

    Let the check run to completion. See the note above about interrupting fsck.gfs2.

  6. If time permits, rerun fsck.gfs2 and verify that no errors are detected. An exit status of 0 means no errors were found, and 1 means errors were detected.

    # fsck.gfs2 -v -y <device>
# echo $?
0

  7. Remount the filesystem. If pacemaker manages it, enable the resource disabled in step 2.

    # pcs resource enable <filesystem_resource>

If issues persist, contact Red Hat Global Support Services and provide the fsck.gfs2 output and the captured metadata.

Example of running fsck.gfs2

# Disable the resource for this Pacemaker configuration.
# pcs resource disable mydata_fs-clone

# On each node, verify the filesystem is not mounted.
# mount | grep /mnt/mydata_lvol1

# (Optional) Capture the metadata first, using the procedure in the article
# "How do I save GFS2 metadata for troubleshooting on Red Hat Enterprise Linux 7, 8, and 9?"

# From one node only, check the filesystem.
# fsck.gfs2 -v -y /dev/mydata/lvol1 2>&1 | tee /root/mydata_lvol1-fsck.log

# Rerun to confirm the filesystem is clean.
# fsck.gfs2 -v -y /dev/mydata/lvol1
# echo $?
0

# Remount the filesystem.
# pcs resource enable mydata_fs-clone

Related Documentation

Root Cause

When a file system error occurs, a gfs2 file system withdraws and becomes unavailable.

  • Filesystem errors can lie dormant in the filesystem for some time before that section of the filesystem is accessed and the withdrawal is triggered.
  • Every node on which the filesystem withdrew must be rebooted before the filesystem can be remounted.
  • Remounting the filesystem before the node is rebooted can result in further corruption, as stale data may remain in memory.
  • The withdrawal was implemented as a gentler option than a kernel panic. It allows the cluster administrator to conduct a post-mortem of the issue and reboot the node at a convenient time. Services on the node that do not use the gfs2 filesystem can continue to run until the reboot.

Diagnostic Steps

If the following symptoms are present, this solution may apply:

  • The gfs2 filesystem becomes unavailable and cannot be accessed.
  • Withdrawal messages are logged in /var/log/messages on one of the cluster nodes.
  • The virtual file /sys/fs/gfs2/<cluster_name>:<fs_name>/withdraw has a value of 1 if the gfs2 filesystem has withdrawn on that node.

Note: The withdraw file and the system log only show whether the filesystem withdrew on that particular node. Other nodes that mount the same filesystem may or may not have withdrawn. Run the checks below on every cluster node.

Run the following as root on the node where the file system became unavailable. A withdrawal is logged only on the node that detected the problem, so check the logs of every node that had the file system mounted.

  1. Check whether a file system is currently withdrawn on the node (run on each node). The file contains 1 if that file system has withdrawn and 0 if it has not. This only shows withdrawals that occurred since the last reboot: the withdrawn file system stays mounted, but unusable, until the node is rebooted, and after a reboot the entry either no longer exists (file system not mounted) or shows 0 again (file system freshly mounted) — a withdrawal from before the reboot is no longer visible. In that case, find the withdrawal in the logs instead (steps 2 and 3):

    # cat /sys/fs/gfs2/<cluster>:<fsname>/withdraw

    To check every gfs2 file system on the node at once:

    # for locktable in $(ls /sys/fs/gfs2/); do printf '%s: %s\n' "$locktable" "$(cat "/sys/fs/gfs2/$locktable/withdraw")"; done

  2. Search the kernel messages for the withdrawal report.
    The current boot, via journald:

    # journalctl -k | grep -iE 'gfs2.*(fatal|withdraw|assertion)'

    Via rsyslog:

    # grep -iE 'gfs2.*(fatal|withdraw|assertion)' /var/log/messages

    To narrow the output to a time window:

    # journalctl -k --since "YYYY-MM-DD HH:MM:SS" --until "YYYY-MM-DD HH:MM:SS"

    If the fsid is known, show every kernel message for that file system:

    # journalctl -k | grep -i 'fsid=<cluster>:<fsname>'
# grep -i 'fsid=<cluster>:<fsname>' /var/log/messages

  3. If the node was rebooted or fenced after the withdrawal, dmesg no longer contains the report. Use the previous boot of the persistent journal (persistent by default on RHEL 7 and later; if /var/log/journal is missing, use /var/log/messages instead):

    # journalctl -k -b -1 | grep -iE 'gfs2.*(fatal|withdraw|assertion)'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments