A GFS2 withdrawal occur because of a "fatal: invalid metadata block" on "function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 350" on RHEL 6, 7

Issue

• A GFS2 withdrawal occur because of a fatal: invalid metadata block on function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 350 on RHEL 6.

Jan 12 17:04:10 node42 kernel: GFS2: fsid=mycluster:sharelv.0: jid=5: Trying to acquire journal lock...
Jan 12 17:04:10 node42 kernel: GFS2: fsid=mycluster:sharelv.0: jid=5: Looking at journal...
Jan 12 17:04:10 node42 kernel: GFS2: fsid=mycluster:sharelv.0: jid=5: Acquiring the transaction lock...
Jan 12 17:04:10 node42 kernel: GFS2: fsid=mycluster:sharelv.0: jid=5: Replaying journal...
Jan 12 17:04:11 node42 kernel: GFS2: fsid=mycluster:sharelv.0: fatal: invalid metadata block
Jan 12 17:04:11 node42 kernel: GFS2: fsid=mycluster:sharelv.0:   bh = 179291 (type: exp=9, found=4)
Jan 12 17:04:11 node42 kernel: GFS2: fsid=mycluster:sharelv.0:   function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 350
Jan 12 17:04:11 node42 kernel: GFS2: fsid=mycluster:sharelv.0: about to withdraw this file system
Jan 12 17:04:12 node42 kernel: GFS2: fsid=mycluster:sharelv.0: telling LM to unmount
Jan 12 17:04:12 node42 kernel: GFS2: fsid=mycluster:sharelv.0: withdrawn
Jan 12 17:04:12 node42 kernel: Pid: 2760, comm: kslowd001 Not tainted 2.6.32-504.8.1.el6.x86_64 #1
Jan 12 17:04:12 node42 kernel: Call Trace:
Jan 12 17:04:12 node42 kernel: [<ffffffffa046f9b8>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa045a041>] ? gfs2_meta_ra+0x71/0x170 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa046fac0>] ? gfs2_metatype_check_ii+0x50/0x60 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa0467456>] ? gfs2_recover_work+0x6e6/0x7b0 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa0466fca>] ? gfs2_recover_work+0x25a/0x7b0 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa0466e2b>] ? gfs2_recover_work+0xbb/0x7b0 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffffa0453939>] ? gfs2_glock_nq_num+0x59/0xa0 [gfs2]
Jan 12 17:04:12 node42 kernel: [<ffffffff81117ef3>] ? slow_work_execute+0x233/0x310
Jan 12 17:04:12 node42 kernel: [<ffffffff81118127>] ? slow_work_thread+0x157/0x360
Jan 12 17:04:12 node42 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
Jan 12 17:04:12 node42 kernel: [<ffffffff81117fd0>] ? slow_work_thread+0x0/0x360
Jan 12 17:04:12 node42 kernel: [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
Jan 12 17:04:12 node42 kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Jan 12 17:04:12 node42 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
Jan 12 17:04:12 node42 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20

• A GFS2 withdrawal occur because of a function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 351 on RHEL 7

Feb 21 14:14:47 node42 clvm(clvmd)[11503]: INFO: 1 logical volume(s) in volume group "vglog" now active 1 logical volume(s) in volume group "vgshared" now active 1 logical volume(s) in volume group "v
gdata" now active 3 logical volume(s) in volume group "vgsystem" now active
Feb 21 14:14:47 node42 clvm(clvmd)[11503]: INFO: PID file (pid:11627 at /var/run/resource-agents/clvmd-clvmd.pid) created for clvmd.
Feb 21 14:14:47 node42 crmd[7956]:   notice: Result of start operation for clvmd on node42-hb: 0 (ok)
Feb 21 14:14:47 node42 Filesystem(clusterfs)[11784]: INFO: Running start for /dev/vgshared/lvshared on /hana/shared
Feb 21 14:14:47 node42 kernel: [  131.551300] GFS2 installed
Feb 21 14:14:47 node42 kernel: [  131.552393] GFS2: fsid=hanacluster:hana-shared: Trying to join cluster "lock_dlm", "hanacluster:hana-shared"
Feb 21 14:14:51 node42 kernel: [  135.373115] GFS2: fsid=hanacluster:hana-shared: Joined cluster. Now mounting FS...
Feb 21 14:14:51 node42 kernel: [  135.395713] GFS2: fsid=hanacluster:hana-shared.1: jid=1, already locked for use
Feb 21 14:14:51 node42 kernel: [  135.395715] GFS2: fsid=hanacluster:hana-shared.1: jid=1: Looking at journal...
Feb 21 14:14:51 node42 kernel: [  135.446477] GFS2: fsid=hanacluster:hana-shared.1: jid=1: Acquiring the transaction lock...
Feb 21 14:14:51 node42 kernel: [  135.446674] GFS2: fsid=hanacluster:hana-shared.1: jid=1: Replaying journal...
Feb 21 14:14:52 node42 kernel: [  135.716079] GFS2: fsid=hanacluster:hana-shared.1: fatal: invalid metadata block
Feb 21 14:14:52 node42 kernel: [  135.716079] GFS2: fsid=hanacluster:hana-shared.1:   bh = 59614 (type: exp=9, found=4)
Feb 21 14:14:52 node42 kernel: [  135.716079] GFS2: fsid=hanacluster:hana-shared.1:   function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 351
Feb 21 14:14:52 node42 kernel: [  135.716084] GFS2: fsid=hanacluster:hana-shared.1: about to withdraw this file system
Feb 21 14:14:52 node42 kernel: [  135.719275] GFS2: fsid=hanacluster:hana-shared.1: telling LM to unmount
Feb 21 14:14:52 node42 kernel: [  135.836933] GFS2: fsid=hanacluster:hana-shared.1: withdrawn
Feb 21 14:14:52 node42 kernel: [  135.836942] CPU: 92 PID: 1593 Comm: kworker/92:1 Tainted: G           OE  ------------   3.10.0-514.6.1.el7.x86_64 #1
Feb 21 14:14:52 node42 kernel: [  135.836944] Hardware name: HITACHI Compute Blade 520XB2/520X B2         , BIOS 09-27 03/09/2016
Feb 21 14:14:52 node42 kernel: [  135.836960] Workqueue: gfs_recovery gfs2_recover_func [gfs2]
Feb 21 14:14:52 node42 kernel: [  135.836964]  ffff8c7d37b9b000 00000000f9db4170 ffff8c7d3d27bbf8 ffffffff816861ac
Feb 21 14:14:52 node42 kernel: [  135.836966]  ffff8c7d3d27bc70 ffffffffa07dbc56 ffff8c7d00000030 ffff8c7d3d27bca8
Feb 21 14:14:52 node42 kernel: [  135.836968]  ffff8c7d3d27bc28 00000000f9db4170 0000000000000000 00000000f9db4170
Feb 21 14:14:52 node42 kernel: [  135.836971] Call Trace:
Feb 21 14:14:52 node42 kernel: [  135.836982]  [<ffffffff816861ac>] dump_stack+0x19/0x1b
Feb 21 14:14:52 node42 kernel: [  135.836988]  [<ffffffffa07dbc56>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Feb 21 14:14:52 node42 kernel: [  135.836993]  [<ffffffffa07dbfa1>] gfs2_metatype_check_ii+0x51/0x60 [gfs2]
Feb 21 14:14:52 node42 kernel: [  135.836998]  [<ffffffffa07d2d0f>] gfs2_recover_func+0x6ef/0x710 [gfs2]
Feb 21 14:14:52 node42 kernel: [  135.837005]  [<ffffffff810336e9>] ? sched_clock+0x9/0x10
Feb 21 14:14:52 node42 kernel: [  135.837010]  [<ffffffff810c7e55>] ? sched_clock_cpu+0x85/0xc0
Feb 21 14:14:52 node42 kernel: [  135.837014]  [<ffffffff81029569>] ? __switch_to+0xd9/0x4c0
Feb 21 14:14:52 node42 kernel: [  135.837019]  [<ffffffffa07d280e>] ? gfs2_recover_func+0x1ee/0x710 [gfs2]
Feb 21 14:14:52 node42 kernel: [  135.837022]  [<ffffffff810a805b>] process_one_work+0x17b/0x470
Feb 21 14:14:52 node42 kernel: [  135.837023]  [<ffffffff810a8e96>] worker_thread+0x126/0x410
Feb 21 14:14:52 node42 kernel: [  135.837025]  [<ffffffff810a8d70>] ? rescuer_thread+0x460/0x460
Feb 21 14:14:52 node42 kernel: [  135.837028]  [<ffffffff810b064f>] kthread+0xcf/0xe0
Feb 21 14:14:52 node42 kernel: [  135.837030]  [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
Feb 21 14:14:52 node42 kernel: [  135.837037]  [<ffffffff81696718>] ret_from_fork+0x58/0x90
Feb 21 14:14:52 node42 kernel: [  135.837039]  [<ffffffff810b0580>] ? kthread_create_on_node+0x140/0x140
Feb 21 14:14:52 node42 kernel: [  135.837058] GFS2: fsid=hanacluster:hana-shared.1: jid=1: Failed
Feb 21 14:14:52 node42 kernel: [  135.837098] GFS2: fsid=hanacluster:hana-shared.1: error recovering my journal: -5
Feb 21 14:14:52 node42 Filesystem(clusterfs)[11784]: ERROR: Couldn't mount filesystem /dev/vgshared/lvshared on /hana/shared
Feb 21 14:14:52 node42 lrmd[7953]:   notice: clusterfs_start_0:11784:stderr [ mount: /dev/mapper/vgshared-lvshared: can't read superblock ]
Feb 21 14:14:52 node42 lrmd[7953]:   notice: clusterfs_start_0:11784:stderr [ ocf-exit-reason:Couldn't mount filesystem /dev/vgshared/lvshared on /hana/shared ]
Feb 21 14:14:52 node42 crmd[7956]:   notice: Result of start operation for clusterfs on node42-hb: 1 (unknown error)
Feb 21 14:14:52 node42 crmd[7956]:   notice: node42-hb-clusterfs_start_0:25 [ mount: /dev/mapper/vgshared-lvshared: can't read superblock\nocf-exit-reason:Couldn't mount filesystem /dev/vgshared/lvshared on /hana/shared\n ]
Feb 21 14:14:52 node42 Filesystem(clusterfs)[11927]: INFO: Running stop for /dev/vgshared/lvshared on /hana/shared
Feb 21 14:14:52 node42 crmd[7956]:   notice: Result of stop operation for clusterfs on node42-hb: 0 (ok)

• A RHEL 7 cluster node had a gfs2 withdrawal when pacemaker tried to mount the filesystem:

Nov  3 13:18:45 node42 Filesystem(gfs2-sasdata)[38534]: INFO: Running start for /dev/mapper/shared-sasdata on /shared/sasdata
Nov  3 13:18:45 node42 kernel: GFS2: fsid=sgrid:sasdata: Trying to join cluster "lock_dlm", "sgrid:sasdata"
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata: Joined cluster. Now mounting FS...
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: jid=2, already locked for use
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: jid=2: Looking at journal...
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: fatal: filesystem consistency error
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2:   inode = 3 65713
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2:   function = jhead_scan, file = fs/gfs2/recovery.c, line = 240
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: about to withdraw this file system
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: telling LM to unmount
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: withdrawn
Nov  3 13:18:47 node42 kernel: CPU: 17 PID: 2038 Comm: kworker/17:2 Not tainted 3.10.0-693.2.2.el7.x86_64 #1
Nov  3 13:18:47 node42 kernel: Hardware name: Dell Inc. PowerEdge M620/0T36VK, BIOS 2.5.4 01/27/2016
Nov  3 13:18:47 node42 kernel: Workqueue: gfs_recovery gfs2_recover_func [gfs2]
Nov  3 13:18:47 node42 kernel: ffff880fe5f16000 00000000a1b083c5 ffff881007ea7b78 ffffffff816a3db1
Nov  3 13:18:47 node42 kernel: ffff881007ea7bf0 ffffffffc077e456 0000002000000030 ffff881007ea7c20
Nov  3 13:18:47 node42 kernel: ffff881007ea7ba8 00000000a1b083c5 ffff881007ea7bc0 ffffffff81235eed
Nov  3 13:18:47 node42 kernel: Call Trace:
Nov  3 13:18:47 node42 kernel: [<ffffffff816a3db1>] dump_stack+0x19/0x1b
Nov  3 13:18:47 node42 kernel: [<ffffffffc077e456>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Nov  3 13:18:47 node42 kernel: [<ffffffff81235eed>] ? __brelse+0x3d/0x50
Nov  3 13:18:47 node42 kernel: [<ffffffffc077e6aa>] gfs2_consist_inode_i+0x5a/0x60 [gfs2]
Nov  3 13:18:47 node42 kernel: [<ffffffffc0774d90>] gfs2_find_jhead+0x140/0x170 [gfs2]
Nov  3 13:18:47 node42 kernel: [<ffffffffc0774f58>] gfs2_recover_func+0x198/0x710 [gfs2]
Nov  3 13:18:47 node42 kernel: [<ffffffff81033619>] ? sched_clock+0x9/0x10
Nov  3 13:18:47 node42 kernel: [<ffffffff810c76f5>] ? sched_clock_cpu+0x85/0xc0
Nov  3 13:18:47 node42 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Nov  3 13:18:47 node42 kernel: [<ffffffff810a881a>] process_one_work+0x17a/0x440
Nov  3 13:18:47 node42 kernel: [<ffffffff810a94e6>] worker_thread+0x126/0x3c0
Nov  3 13:18:47 node42 kernel: [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
Nov  3 13:18:47 node42 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Nov  3 13:18:47 node42 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Nov  3 13:18:47 node42 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Nov  3 13:18:47 node42 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: jid=2: Failed
Nov  3 13:18:47 node42 kernel: GFS2: fsid=sgrid:sasdata.2: error recovering my journal: -5
Nov  3 13:18:47 node42 Filesystem(gfs2-sasdata)[38534]: ERROR: Couldn't mount filesystem /dev/mapper/shared-sasdata on /shared/sasdata
Nov  3 13:18:47 node42 lrmd[2467]:  notice: gfs2-sasdata_start_0:38534:stderr [ mount: /dev/mapper/shared-sasdata: can't read superblock ]