A GFS2 withdraw because of an `invalid metadata block` occurred and `fsck.gfs2` failed to fix the filesystem: `Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: fatal: invalid metadata block`

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)
  • A Global Filesystem 2(GFS2)

Issue

  • A gfs2 withdraw occurred after growing a gfs2 filesystem on VMware VM's shared virtual disk. In addition, fsck.gfs2 was unable to fix the gfs2 filesystem:

    Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0: fatal: invalid metadata block
Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0:   bh = 14115477 (magic number)
Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0:   function = gfs2_rgrp_bh_get, file = fs/gfs2/rgrp.c, line = 1123
Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0: about to withdraw this file system
Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0: telling LM to unmount
Mar 27 22:00:04 node43 kernel: GFS2: fsid=mygfs2fs:gfs2_storage2.0: withdrawn

  • A couple of gfs2 withdraws occurred because of an invalid metadata block occurred and fsck.gfs2 failed to fix the filesystem.

    Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: fatal: invalid metadata block
Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0:   bh = 720909 (magic number)
Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0:   function = gfs2_rgrp_bh_get, file = fs/gfs2/rgrp.c, line = 1137
Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: about to withdraw this file system
Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: dirty_inode: glock -5
Oct 27 09:27:53 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: dirty_inode: glock -5

Oct 27 09:47:21 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: fatal: invalid metadata block
Oct 27 09:47:21 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0:   bh = 720102 (magic number)
Oct 27 09:47:21 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 365
Oct 27 09:47:21 node42 kernel: GFS2: fsid=mygfs2fs:shared_home.0: about to withdraw this file system

Oct 27 09:53:36 node42 fsck.gfs2[6463]: started: /dev/mapper/vg_shared_home-lv_shared_home
Oct 27 09:53:36 node42 fsck.gfs2[6463]: exit: 16

Resolution

The GFS2 had to be fixed manually by engineer as the fsck.gfs2 encountered a fatal error. The reason for the corruption is being investigated in bugzilla: Bug 1279994 – A GFS2 "invalid metadata block" withdrawal occurred: function = gfs2_rgrp_bh_get, file = fs/gfs2/rgrp.c, line = 1137

Please contact Red Hat support if a withdrawal similar to the one in the Issues section occurs.

There is another withdrawal that is similar but caused by different issue.

Root Cause

The reason that the withdrawal occurred was because of a couple corrupt resource groups.

Diagnostic Steps

  • Capture an sosreport from all cluster nodes and review the /var/log/messages file from any withdraws that have occurred.
  • Save the metadata for the GFS2 filesystem before fsck.gfs2 is ran.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.