A GFS2 withdrawal occurred: fatal: filesystem consistency error, function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109

Issue

• A storage device fails with some scsi error (or cluster node was fenced with fence_scsi and then a gfs2 filesystem withdraws:

Jul  6 10:15:15 node42 kernel: [506204.622266] GFS2: fsid=cluster1:gfs2fs.1: Error -52 writing to log
[....]
Jul  6 10:15:45 node42 kernel: [506234.682578] GFS2: fsid=cluster1:gfs2fs.1: fatal: invalid metadata block
Jul  6 10:15:45 node42 kernel: [506234.682578] GFS2: fsid=cluster1:gfs2fs.1:   bh = 2039836 (magic number)
Jul  6 10:15:45 node42 kernel: [506234.682578] GFS2: fsid=cluster1:gfs2fs.1:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 365
Jul  6 10:15:45 node42 kernel: [506234.682582] GFS2: fsid=cluster1:gfs2fs.1: about to withdraw this file system
Jul  6 10:15:45 node42 kernel: [506234.685809] device-mapper: multipath: Failing path 66:112.
Jul  6 10:15:45 node42 kernel: [506234.685817] device-mapper: multipath: Failing path 8:48.
Jul  6 10:15:45 node42 kernel: [506234.685809] device-mapper: multipath: Failing path 66:112.
Jul  6 10:15:45 node42 kernel: [506234.685817] device-mapper: multipath: Failing path 8:48.
Jul  6 10:15:45 node42 kernel: [506234.689373] sd 12:0:0:3: alua: port group 02 state N preferred supports tolUsNA
Jul  6 10:15:45 node42 kernel: [506234.690193] sd 12:0:0:3: alua: port group 02 switched to state A
Jul  6 10:15:45 node42 kernel: [506234.690288] sd 0:0:1:3: Parameters changed
Jul  6 10:15:45 node42 kernel: [506234.695854] sd 0:0:1:3: alua: port group 02 state A preferred supports tolUsNA
Jul  6 10:15:45 node42 kernel: [506234.697790] GFS2: fsid=cluster1:gfs2fs.1: telling LM to unmount
Jul  6 10:15:45 node42 kernel: [506234.699130] GFS2: fsid=cluster1:gfs2fs.1: withdrawn
Jul  6 10:15:45 node42 kernel: [506234.699528] CPU: 0 PID: 5104 Comm: db2sysc Not tainted 3.10.0-327.13.1.el7.x86_64 #1
Jul  6 10:15:45 node42 kernel: [506234.699529] Hardware name: Cisco Systems Inc UCSC-C240-M4S/UCSC-C240-M4S, ....
Jul  6 10:15:45 node42 kernel: [506234.699530]  ffff882004653000 00000000ee8b7a21 ffff880023a8bc08 ffffffff816356f4
Jul  6 10:15:45 node42 kernel: [506234.699534]  ffff880023a8bc80 ffffffffa0824c91 0000000000000030 ffff880023a8bcb0
Jul  6 10:15:45 node42 kernel: [506234.699549]  ffff880023a8bc38 00000000ee8b7a21 ffff880023a8bc38 ffff880023a8bc38
Jul  6 10:15:45 node42 kernel: [506234.699552] Call Trace:
Jul  6 10:15:45 node42 kernel: [506234.699558]  [<ffffffff816356f4>] dump_stack+0x19/0x1b
Jul  6 10:15:45 node42 kernel: [506234.699568]  [<ffffffffa0824c91>] gfs2_lm_withdraw+0x141/0x170 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699573]  [<ffffffffa0824f75>] gfs2_meta_check_ii+0x45/0x50 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699577]  [<ffffffffa080c868>] gfs2_meta_indirect_buffer+0xd8/0x140 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699581]  [<ffffffffa0822cbd>] gfs2_dirty_inode+0x11d/0x250 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699585]  [<ffffffffa0822c78>] ? gfs2_dirty_inode+0xd8/0x250 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699593]  [<ffffffff811759a5>] ? do_writepages+0x35/0x40
Jul  6 10:15:45 node42 kernel: [506234.699653]  [<ffffffff8120b61a>] __mark_inode_dirty+0xca/0x290
Jul  6 10:15:45 node42 kernel: [506234.699659]  [<ffffffffa0820906>] gfs2_write_inode+0xb6/0xe0 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699661]  [<ffffffff81208587>] __writeback_single_inode+0x167/0x220
Jul  6 10:15:45 node42 kernel: [506234.699664]  [<ffffffff81209247>] writeback_single_inode+0xe7/0x1b0
Jul  6 10:15:45 node42 kernel: [506234.699666]  [<ffffffff8120934e>] sync_inode+0x3e/0x60
Jul  6 10:15:45 node42 kernel: [506234.699668]  [<ffffffff812093b1>] sync_inode_metadata+0x41/0x60
Jul  6 10:15:45 node42 kernel: [506234.699675]  [<ffffffffa080f730>] gfs2_fsync+0xb0/0x120 [gfs2]
Jul  6 10:15:45 node42 kernel: [506234.699679]  [<ffffffff8120f9c5>] do_fsync+0x65/0xa0
Jul  6 10:15:45 node42 kernel: [506234.699682]  [<ffffffff8120fc90>] SyS_fsync+0x10/0x20
Jul  6 10:15:45 node42 kernel: [506234.699130] GFS2: fsid=cluster1:gfs2fs.1: withdrawn

• A scsi error occurred right before a gfs2 withdrawal:

May  2 23:51:01 node42 kernel: sd 3:0:2:0: [sdf] Unhandled error code
May  2 23:51:01 node42 kernel: sd 3:0:2:0: [sdf] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May  2 23:51:01 node42 kernel: sd 3:0:2:0: [sdf] CDB: Write(10): 2a 00 00 c3 20 d8 00 00 08 00
May  2 23:51:01 node42 kernel: Buffer I/O error on device dm-11, logical block 1598235
May  2 23:51:01 node42 kernel: lost page write due to I/O error on dm-11
May  2 23:53:22 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1: fatal: filesystem consistency error
May  2 23:53:22 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1:   RG = 1965752
May  2 23:53:22 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1:   function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109
May  2 23:53:22 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1: about to withdraw this file system
May  2 23:53:24 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1: telling LM to unmount
May  2 23:53:24 node42 kernel: GFS2: fsid=sasgrid:sasconfig.1: withdrawn