A gfs2 withdrawal occurred: fatal: invalid metadata block#012 bh = 1261644 (magic number)#012 function = get_leaf, file = fs/gfs2/dir.c, line = 767

Issue

• A gfs2 withdrawal occurred:

   Jul 12 08:46:37 node42 kernel: gfs2: fsid=cluster3:data.1: fatal: invalid metadata block
   Jul 12 08:46:37 node42 kernel: gfs2: bh = 1261644 (magic number)  
   Jul 12 08:46:37 node42 kernel: gfs2: function = get_leaf, file = fs/gfs2/dir.c, line = 767
   Jul 12 08:46:37 node42 kernel: gfs2: fsid=cluster3:data.1: about to withdraw this file system
   Jul 12 08:46:43 node42 kernel: gfs2: fsid=cluster3:data.1: Requesting recovery of jid 1.
   Jul 12 08:46:43 node42 kernel: gfs2: fsid=cluster3:data.1: No other mounters found. Trying to recover our own journal jid 1.
   Jul 12 08:46:43 node42 kernel: gfs2: fsid=cluster3:data.1: jid=1: Recovery not attempted due to withdraw.
   Jul 12 08:46:43 node42 kernel: gfs2: fsid=cluster3:data.1: recovery_result jid 1 ignored due to withdraw.
   Jul 12 08:46:44 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:45 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:46 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:47 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:49 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:50 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:51 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:52 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:53 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:54 node42 kernel: gfs2: fsid=cluster3:data.1: Waiting for journal recovery jid 1.
   Jul 12 08:46:54 node42 kernel: gfs2: fsid=cluster3:data.1: Journal recovery skipped for 1 until next mount.
   Jul 12 08:46:54 node42 kernel: gfs2: fsid=cluster3:data.1: Glock dequeues delayed: 0
   Jul 12 08:46:54 node42 kernel: gfs2: fsid=cluster3:data.1: telling LM to unmount
   Jul 12 08:46:54 node42 kernel: dlm: gfs2-udfs: leaving the lockspace group...
   Jul 12 08:46:54 node42 kernel: dlm: gfs2-udfs: group event done -512 0
   Jul 12 08:46:54 node42 kernel: dlm: gfs2-udfs: group leave failed -512 0
   Jul 12 08:46:54 node42 kernel: dlm: gfs2-udfs: release_lockspace final free
   Jul 12 08:46:54 node42 kernel: gfs2: fsid=cluster3:data.1: File system withdrawn
   Jul 12 08:46:54 node42 kernel: CPU: 2 PID: 1365468 Comm: find Kdump: loaded Not tainted 4.18.0-372.19.1.el8_6.x86_64 #1
   Jul 12 08:46:54 node42 kernel: Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
   Jul 12 08:46:54 node42 kernel: Call Trace:
   Jul 12 08:46:54 node42 kernel: dump_stack+0x41/0x60
   Jul 12 08:46:54 node42 kernel: gfs2_withdraw.cold.16+0xd9/0x4cc [gfs2]
   Jul 12 08:46:54 node42 kernel: gfs2_meta_check_ii+0x2f/0x40 [gfs2]
   Jul 12 08:46:54 node42 kernel: get_leaf+0xa5/0xb0 [gfs2]
   Jul 12 08:46:54 node42 kernel: gfs2_dir_read+0x3ad/0x660 [gfs2]
   Jul 12 08:46:54 node42 kernel: gfs2_readdir+0x6d/0xb0 [gfs2]
   Jul 12 08:46:54 node42 kernel: ? gfs2_readdir+0x4e/0xb0 [gfs2]
   Jul 12 08:46:54 node42 kernel: iterate_dir+0x13c/0x190
   Jul 12 08:46:54 node42 kernel: ? f_dupfd+0x62/0x74
   Jul 12 08:46:54 node42 kernel: ksys_getdents64+0x9c/0x130
   Jul 12 08:46:54 node42 kernel: ? iterate_dir+0x190/0x190
   Jul 12 08:46:54 node42 kernel: __x64_sys_getdents64+0x16/0x20
   Jul 12 08:46:54 node42 kernel: do_syscall_64+0x5b/0x1a0
   Jul 12 08:46:54 node42 kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
   Jul 12 08:46:54 node42 kernel: RIP: 0033:0x7f79ad0d615b
   Jul 12 08:46:54 node42 kernel: Code: 00 00 48 83 c4 08 5b 5d c3 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 f9 8c 2c 00 f7 d8
   Jul 12 08:46:54 node42 kernel: RSP: 002b:00007ffea47bf628 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9
   Jul 12 08:46:54 node42 kernel: RAX: ffffffffffffffda RBX: 00005640c8a8f540 RCX: 00007f79ad0d615b
   Jul 12 08:46:54 node42 kernel: RDX: 0000000000008000 RSI: 00005640c8a8f570 RDI: 0000000000000006
   Jul 12 08:46:54 node42 kernel: RBP: 00005640c8a8f570 R08: 00005640c8a84364 R09: 0000000000000000
   Jul 12 08:46:54 node42 kernel: R10: 0000000000000100 R11: 0000000000000246 R12: fffffffffffffe98
   Jul 12 08:46:54 node42 kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 00005640c8a8f540
   Jul 12 08:46:54 node42 dlm_controld[404115]: 191443 open "/sys/kernel/dlm/gfs2-udfs/control" error -1 2
   Jul 12 08:46:54 node42 dlm_controld[404115]: 191443 open "/sys/kernel/dlm/gfs2-udfs/event_done" error -1 2