A RHEL 6 GFS2 filesystem had a withdraw: "fatal: filesystem consistency error" in "function = gfs2_replay_read_block, file = fs/gfs2/recovery.c, line = 45"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)
  • Red Hat Enterprise Linux Server 7 (with the High Availability and Resilient Storage Add Ons)
  • GFS2 (Global Filesystem 2)

Issue

A RHEL 6 GFS2 filesystem had a withdraw while replaying the journal when they tried to mount the filesystem:

Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: jid=5: Trying to acquire journal lock...
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: jid=5: Looking at journal...
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: jid=5: Acquiring the transaction lock...
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: jid=5: Replaying journal...
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: fatal: filesystem consistency error
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0:   inode = 17 116517
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0:   function = gfs2_replay_read_block, file = fs/gfs2/recovery.c, line = 45
Feb  3 18:43:46 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: about to withdraw this file system
Feb  3 18:43:47 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: telling LM to unmount
Feb  3 18:43:47 node42 kernel: GFS2: fsid=gfs2clus:lv_arrakis.0: withdrawn

Resolution

There is a known issue with GFS2 replaying journals of different sizes which is resolved with the following errata:

Workaround:
If you have journals of different sizes then it is recommended that the filesystem be recreated with journals of the same size of 128MBs. There is no way to modify the size of existing journal and reason that we recommend that the filesystem be recreated.

Root Cause

If you used gfs2_jadd to add new journals of a size smaller than the existing journals, replaying those new journals would withdraw. That's because function gfs2_replay_incr_blk was using the number of journal blocks (jd_block) from the superblock's journal pointer. In other words, "My journal's max size" rather than "the journal we're replaying's size." This patch changes the function to use the size of the pertinent journal rather than always using the journal we happen to be using.

Diagnostic Steps

  • Check the logs to see if the withdraw is similar to the one in the "Issues" section. If the withdraw is similar then you are affected by this issue.
  • Was a journal added to the gfs2 filesystem?
  • Are there any journals on the gfs2 filesystem that are less than 128MB in size?
# gfs2_tool journals <mount point for gfs2 filesystem>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.