A RHEL 6, 7 gfs2 filesystem had a withdraw: function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109

Issue

• A withdrawal occurred on a RHEL 7 gfs2 filesystem shortly after it was mounted at boot time:

[ 1163.633594] GFS2 installed
[ 1163.634124] GFS2: fsid=rh7_cluster:mydata: Trying to join cluster "lock_dlm", "rh7_cluster:mydata"
[ 1164.138968] GFS2: fsid=rh7_cluster:mydata: first mounter control generation 0
[ 1164.138970] GFS2: fsid=rh7_cluster:mydata: Joined cluster. Now mounting FS...
[ 1164.168104] GFS2: fsid=rh7_cluster:mydata.0: jid=0, already locked for use
[ 1164.168106] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Looking at journal...
[ 1164.201735] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Acquiring the transaction lock...
[ 1164.201768] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Replaying journal...
[ 1164.230452] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Replayed 39 of 47 blocks
[ 1164.230454] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Found 5 revoke tags
[ 1164.230860] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Journal replayed in 1s
[ 1164.230875] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Done
[ 1164.230912] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Trying to acquire journal lock...
[ 1164.231893] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Looking at journal...
[ 1164.287488] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Acquiring the transaction lock...
[ 1164.287523] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Replaying journal...
[ 1164.290579] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Replayed 39 of 47 blocks
[ 1164.290581] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Found 5 revoke tags
[ 1164.291151] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Journal replayed in 1s
[ 1164.291168] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Done
[ 1164.291203] GFS2: fsid=rh7_cluster:mydata.0: first mount done, others may mount
[.....]
[ 1187.852687] GFS2: buf_blk = 0x417 old_state=0, new_state=2
[ 1187.852690] GFS2: rgrp=0x1f0005 bi_start=0x0
[ 1187.852691] GFS2: bi_offset=0x80 bi_len=0xf80
[ 1187.852694] CPU: 0 PID: 12055 Comm: db2start Not tainted 3.10.0-327.13.1.el7.x86_64 #1
[ 1187.852695] Hardware name: Cisco Systems Inc UCSC-C240-M4S/UCSC-C240-M4S, BIOS C240M4.2.0.4a.0.042220151400 04/22/2015
[ 1187.852697]  00000000001f0422 00000000756d825d ffff881010a0fc80 ffffffff816356f4
[ 1187.852702]  ffff881010a0fce0 ffffffffa07705e8 00000002577d53a5 ffff881ff76ab110
[ 1187.852705]  ffff88201f6af200 0000000000000417 00000000756d825d ffff881005260dc0
[ 1187.852709] Call Trace:
[ 1187.852718]  [<ffffffff816356f4>] dump_stack+0x19/0x1b
[ 1187.852731]  [<ffffffffa07705e8>] rgblk_free+0x1f8/0x230 [gfs2]
[ 1187.852736]  [<ffffffffa0772d67>] gfs2_unlink_di+0x37/0x100 [gfs2]
[ 1187.852742]  [<ffffffffa077a165>] gfs2_unlink_inode+0x6e/0x76 [gfs2]
[ 1187.852747]  [<ffffffffa0767479>] gfs2_unlink+0x219/0x220 [gfs2]
[ 1187.852752]  [<ffffffffa07672f0>] ? gfs2_unlink+0x90/0x220 [gfs2]
[ 1187.852756]  [<ffffffffa0767318>] ? gfs2_unlink+0xb8/0x220 [gfs2]
[ 1187.852759]  [<ffffffffa076734c>] ? gfs2_unlink+0xec/0x220 [gfs2]
[ 1187.852764]  [<ffffffff811eaf41>] vfs_unlink+0xe1/0x150
[ 1187.852767]  [<ffffffff811ef44e>] do_unlinkat+0x26e/0x2b0
[ 1187.852771]  [<ffffffff811e060e>] ? ____fput+0xe/0x10
[ 1187.852776]  [<ffffffff810a22bc>] ? task_work_run+0xac/0xe0
[ 1187.852780]  [<ffffffff811f0396>] SyS_unlink+0x16/0x20
[ 1187.852784]  [<ffffffff81645e89>] system_call_fastpath+0x16/0x1b
[ 1187.852799] GFS2: fsid=rh7_cluster:mydata.0: fatal: filesystem consistency error
               GFS2: fsid=rh7_cluster:mydata.0:   RG = 2031621
               GFS2: fsid=rh7_cluster:mydata.0:   function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109
[ 1187.852818] GFS2: fsid=rh7_cluster:mydata.0: about to withdraw this file system
[ 1187.864039] GFS2: fsid=rh7_cluster:mydata.0: telling LM to unmount
[ 1187.864507] GFS2: fsid=rh7_cluster:mydata.0: withdrawn

• A withdrawal occurred on a RHEL 7 gfs2 filesystem:

Jul 31 11:41:06 node42 kern.warning kernel:GFS2: fsid=mycluster:share.0: fatal: filesystem consistency error
Jul 31 11:41:06 node42 GFS2: fsid=mycluster:share.0:   RG = 202579796
Jul 31 11:41:06 node42 GFS2: fsid=mycluster:share.0:   function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109
Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: about to withdraw this file system
Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: telling LM to unmount
Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: withdrawn
Jul 31 11:41:06 node42 kern.warning kernel:CPU: 3 PID: 11249 Comm: java Not tainted 3.10.0-514.2.2.el7.x86_64 #1
Jul 31 11:41:06 node42 kern.warning kernel:Hardware name: Dell Inc. PowerEdge M620/0VHRN7, BIOS 1.7.6 05/23/2013
Jul 31 11:41:06 node42 kern.warning kernel: ffff881fb8176000 0000000078f8a0ae ffff8818d9fff800 ffffffff816860cc
Jul 31 11:41:06 node42 kern.warning kernel: ffff8818d9fff878 ffffffffa0ab5c56 ffff881800000030 ffff8818d9fff8a0
Jul 31 11:41:06 node42 kern.warning kernel: ffff8818d9fff830 0000000078f8a0ae ffff881fbee43fc0 ffff8818d9fff8e0
Jul 31 11:41:06 node42 kern.warning kernel:Call Trace:
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff816860cc>] dump_stack+0x19/0x1b
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0ab5c56>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0ab5ef7>] gfs2_consist_rgrpd_i+0x47/0x50 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0aae337>] rgblk_free+0x217/0x230 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0ab0914>] __gfs2_free_blocks+0x34/0x120 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a89247>] recursive_scan+0x6c7/0x6f0 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff8118020e>] ? __find_get_page+0x1e/0xa0
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a8938a>] trunc_dealloc+0x11a/0x140 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff81200000>] ? __fput+0x80/0x260
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a8aa03>] do_shrink+0x1e3/0x550 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a8aff3>] gfs2_setattr_size+0x173/0x270 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a97d90>] ? gfs2_glock_nq+0x260/0x450 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0aa7bbf>] gfs2_setattr+0x11f/0x2b0 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0a95948>] ? gfs2_glock_add_to_lru+0x68/0x80 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffffa0aa7b18>] ? gfs2_setattr+0x78/0x2b0 [gfs2]
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff8121b879>] notify_change+0x279/0x3d0
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff812c98fe>] ? process_measurement+0x8e/0x250
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff811fc3f5>] do_truncate+0x75/0xc0
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff8120d062>] do_last+0x5f2/0x12a0
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff8120ddd2>] path_openat+0xc2/0x490
Jul 31 11:41:06 node42 kern.warning kernel: [<ffffffff811806bb>] ? unlock_page+0x2b/0x30