A gfs2 filesystem withdraws with the error: function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 230

Issue

• A gfs2 filesystem withdraws with the following error and call trace:

Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: journal 5 mapped with 90 extents in 0ms
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: jid=5: Journal head lookup took 581ms
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: jid=5: Acquiring the transaction lock...
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: jid=5: Replaying journal...
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: fatal: invalid metadata block
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0:   bh = 338867 (type: exp=9, found=2)
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0:   function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 230
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: about to withdraw this file system
Apr 26 04:36:52 node42 kernel: CPU: 5 PID: 511 Comm: kworker/5:2 Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Apr 26 04:36:52 node42 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
Apr 26 04:36:52 node42 kernel: Workqueue: gfs_recovery gfs2_recover_func [gfs2]
Apr 26 04:36:52 node42 kernel: Call Trace:
Apr 26 04:36:52 node42 kernel: [<ffffffffb7f7ff85>] dump_stack+0x19/0x1b
Apr 26 04:36:52 node42 kernel: [<ffffffffc0555396>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffc0555711>] gfs2_metatype_check_ii+0x51/0x60 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffc054bfb7>] gfs2_recover_func+0x7e7/0x860 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffc05362b3>] ? gfs2_glock_nq_num+0x73/0xb0 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffb782b59e>] ? __switch_to+0xce/0x580
Apr 26 04:36:52 node42 kernel: [<ffffffffc054b91f>] ? gfs2_recover_func+0x14f/0x860 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffc054ba8d>] ? gfs2_recover_func+0x2bd/0x860 [gfs2]
Apr 26 04:36:52 node42 kernel: [<ffffffffb78be6bf>] process_one_work+0x17f/0x440
Apr 26 04:36:52 node42 kernel: [<ffffffffb78bf7d6>] worker_thread+0x126/0x3c0
Apr 26 04:36:52 node42 kernel: [<ffffffffb78bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 26 04:36:52 node42 kernel: [<ffffffffb78c6691>] kthread+0xd1/0xe0
Apr 26 04:36:52 node42 kernel: [<ffffffffb78c65c0>] ? insert_kthread_work+0x40/0x40
Apr 26 04:36:52 node42 kernel: [<ffffffffb7f92d37>] ret_from_fork_nospec_begin+0x21/0x21
Apr 26 04:36:52 node42 kernel: [<ffffffffb78c65c0>] ? insert_kthread_work+0x40/0x40
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: jid=5: Failed
Apr 26 04:36:52 node42 kernel: GFS2: fsid=cluster3:mydata.0: error recovering journal 5: -5