A RHEL 6 GFS2 filesystem is withdrawing because of "invalid metadata block" or "filesystem consistency error"

Issue

• A RHEL 6 GFS2 filesystem is withdrawing because of invalid metadata block or filesystem consistency error. Below is a couple of the common backtraces that occur:

GFS2: fsid=intec_cluster:scratch.1: jid=1, already locked for use
GFS2: fsid=intec_cluster:scratch.1: jid=1: Looking at journal...
GFS2: fsid=intec_cluster:scratch.1: jid=1: Done
GFS2: buf_blk = 0xbd9 old_state=0, new_state=0
GFS2: rgrp=0x11 bi_start=0x3f38
GFS2: bi_offset=0x18 bi_len=0x8c2
Pid: 45635, comm: bobs_accordion4 Not tainted 2.6.32-431.el6.x86_64 #1
Call Trace:
 [<ffffffffa059c816>] ? rgblk_free+0x1d6/0x200 [gfs2]
 [<ffffffffa059cae1>] ? __gfs2_free_blocks+0x41/0xf0 [gfs2]
 [<ffffffffa059caa1>] ? __gfs2_free_blocks+0x1/0xf0 [gfs2]
 [<ffffffffa057522a>] ? recursive_scan+0x6fa/0x740 [gfs2]
 [<ffffffffa0575355>] ? trunc_dealloc+0xe5/0x100 [gfs2]
 [<ffffffffa0581b0f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]
 [<ffffffff8118ae1d>] ? __sb_end_write+0x3d/0x70
 [<ffffffffa05a0b87>] ? gfs2_trans_end+0xb7/0x1b0 [gfs2]
 [<ffffffffa0576bdb>] ? do_shrink+0x1eb/0x550 [gfs2]
 [<ffffffff8109b320>] ? wake_bit_function+0x0/0x50
 [<ffffffffa057707b>] ? gfs2_setattr_size+0x10b/0x240 [gfs2]
 [<ffffffffa0593b9c>] ? gfs2_setattr+0x20c/0x320 [gfs2]
 [<ffffffffa0581b0f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]
 [<ffffffffa05939d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffffa0582bae>] ? gfs2_glock_dq_uninit+0x1e/0x30 [gfs2]
 [<ffffffff811a7338>] ? notify_change+0x168/0x340
 [<ffffffff81187064>] ? do_truncate+0x64/0xa0
 [<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff8119bcb1>] ? do_filp_open+0x861/0xd20
 [<ffffffff81282705>] ? _atomic_dec_and_lock+0x55/0x80
 [<ffffffff811a8212>] ? alloc_fd+0x92/0x160
 [<ffffffff81185d29>] ? do_sys_open+0x69/0x140
 [<ffffffff8100c715>] ? math_state_restore+0x45/0x60
 [<ffffffff81185e40>] ? sys_open+0x20/0x30
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
GFS2: fsid=intec_cluster:scratch.1: fatal: filesystem consistency error
GFS2: fsid=intec_cluster:scratch.1:   RG = 17
GFS2: fsid=intec_cluster:scratch.1:   function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 103
GFS2: fsid=intec_cluster:scratch.1: about to withdraw this file system
GFS2: fsid=intec_cluster:scratch.1: telling LM to unmount
GFS2: fsid=intec_cluster:scratch.1: withdrawn
Pid: 45635, comm: bobs_accordion4 Not tainted 2.6.32-431.el6.x86_64 #1
Call Trace:
 [<ffffffffa05a14a8>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
 [<ffffffff81010eb5>] ? show_trace_log_lvl+0x55/0x70
 [<ffffffffa05a1657>] ? gfs2_consist_rgrpd_i+0x47/0x50 [gfs2]
 [<ffffffffa059c835>] ? rgblk_free+0x1f5/0x200 [gfs2]
 [<ffffffffa059cae1>] ? __gfs2_free_blocks+0x41/0xf0 [gfs2]
 [<ffffffffa059caa1>] ? __gfs2_free_blocks+0x1/0xf0 [gfs2]
 [<ffffffffa057522a>] ? recursive_scan+0x6fa/0x740 [gfs2]
 [<ffffffffa0575355>] ? trunc_dealloc+0xe5/0x100 [gfs2]
 [<ffffffffa0581b0f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]
 [<ffffffff8118ae1d>] ? __sb_end_write+0x3d/0x70
 [<ffffffffa05a0b87>] ? gfs2_trans_end+0xb7/0x1b0 [gfs2]
 [<ffffffffa0576bdb>] ? do_shrink+0x1eb/0x550 [gfs2]
 [<ffffffff8109b320>] ? wake_bit_function+0x0/0x50
 [<ffffffffa057707b>] ? gfs2_setattr_size+0x10b/0x240 [gfs2]
 [<ffffffffa0593b9c>] ? gfs2_setattr+0x20c/0x320 [gfs2]
 [<ffffffffa0581b0f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]
 [<ffffffffa05939d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffffa0582bae>] ? gfs2_glock_dq_uninit+0x1e/0x30 [gfs2]
 [<ffffffff811a7338>] ? notify_change+0x168/0x340
 [<ffffffff81187064>] ? do_truncate+0x64/0xa0
 [<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff8119bcb1>] ? do_filp_open+0x861/0xd20
 [<ffffffff81282705>] ? _atomic_dec_and_lock+0x55/0x80
 [<ffffffff811a8212>] ? alloc_fd+0x92/0x160
 [<ffffffff81185d29>] ? do_sys_open+0x69/0x140
 [<ffffffff8100c715>] ? math_state_restore+0x45/0x60
 [<ffffffff81185e40>] ? sys_open+0x20/0x30
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b