Skip to navigation

Why is my GFS/GFS2 filesystem failing to mount after an interrupted or failed fsck attempt?

Updated 2013-12-12T21:20:19+00:00

Issue

  • In order to fsck a GFS or GFS2 filesystem, that filesystem must not be mounted on any node in the cluster while the filesystem check is in progress.  If a node were to mount the filesystem in the middle of an fsck it could lead to filesystem and data corruption.  For this reason, gfs_fsck and gfs2_fsck will change the locking protocol in the superblock to fsck_dlm (which is an invalid protocol string) before beginning the check, and will change it back to its original value (such as lock_dlm) once it is done.  Since this is an invalid string, any node that attempts to mount it during the fsck will be unable to, thereby protecting it from corruption.  If for some reason the fsck operation were to fail unexpectedly (for example from a segmentation fault) or if it were killed by the user (for instance if it were appearing to hang), then this locking protocol string does not get changed back automatically and thus further attempts to mount it will fail with an error such as:

    $ mount -t gfs2 -o noatime,nodiratime,noquota /dev/clust/gfs2-a /mnt/gfs2-a/
    /sbin/mount.gfs2: error mounting /dev/mapper/clust-gfs2--a on /mnt/gfs2-a: No such file or directory
    
  • And in the logs a message such as the following may be seen:

    Jun 19 14:59:35 node1 kernel: GFS2: fsid=: Trying to join cluster "fsck_dlm", "appCluster:gfs2-a"
    Jun 19 14:59:36 node1 kernel: GFS2: can't find protocol fsck_dlm
    Jun 19 14:59:36 node1 kernel: GFS2: fsid=: can't mount proto=fsck_dlm, table=appCluster:gfs2-a, hostdata=
    

Note: The error messages in this example are for GFS2, but are similar for GFS.

Environment

  • Red Hat Cluster Suite 4+ and Red Hat GFS 6.1:
  • Red Hat Enterprise Linux Server 5 (with the High Availability and Resilient Storage Add Ons)

Subscriber content preview. For full access to the Red Hat Knowledgebase, please log in.

Not a subscriber? Learn more about the benefits of Red Hat Subscriptions.