A gfs2 withdrawal occurred: fatal: assertion "!tr->tr_num_buf_new && !tr->tr_num_databuf_new" failed

Issue

• A gfs2 withdrawal occurred:

  [1398558.804143] sd 1:0:1:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
  [1657690.443879] gfs2: fsid=MyCluster:gfs_fs.0: fatal: assertion "!tr->tr_num_buf_new && !tr->tr_num_databuf_new" failed
     function = gfs2_log_flush, file = fs/gfs2/log.c, line = 1010
  [1657690.443951] CPU: 14 PID: 1346152 Comm: kworker/14:1H Kdump: loaded Not tainted 4.18.0-513.24.1.el8_9.x86_64 #1
  [1657690.443954] Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.21805430.B64.2305221830 05/22/2023
  [1657690.443956] Workqueue: glock_workqueue glock_work_func [gfs2]
  [1657690.443980] Call Trace:
  [1657690.443983]  dump_stack+0x41/0x60
  [1657690.443988]  gfs2_log_flush+0xb3d/0xbf0 [gfs2]
  [1657690.444000]  ? sched_clock_cpu+0xc/0xb0
  [1657690.444005]  inode_go_sync+0x80/0x190 [gfs2]
  [1657690.444012]  do_xmote+0xd1/0x340 [gfs2]
  [1657690.444025]  ? __switch_to+0x10c/0x430
  [1657690.444029]  glock_work_func+0x5c/0x110 [gfs2]
  [1657690.444037]  process_one_work+0x1d3/0x390
  [1657690.444040]  ? process_one_work+0x390/0x390
  [1657690.444042]  worker_thread+0x30/0x390
  [1657690.444045]  ? process_one_work+0x390/0x390
  [1657690.444047]  kthread+0x134/0x150
  [1657690.444049]  ? set_kthread_struct+0x50/0x50
  [1657690.444051]  ret_from_fork+0x1f/0x40
  [1657690.444089] gfs2: fsid=MyCluster:gfs_fs.0: about to withdraw this file system
  [1657690.444095] gfs2: fsid=MyCluster:gfs_fs.0: GFS2:adding buf while withdrawn! 0x47f484d
  [1657690.444113] gfs2: fsid=MyCluster:gfs_fs.0: GFS2:adding buf while withdrawn! 0x47f484e
  [1657690.444142] gfs2: fsid=MyCluster:gfs_fs.0: dirty_inode: gfs2_trans_begin -30
  [1657696.932535] gfs2: fsid=MyCluster:gfs_fs.0: Requesting recovery of jid 0.
  [1657697.442908] gfs2: fsid=MyCluster:gfs_fs.0: Journal recovery complete for jid 0.
  [1657697.442911] gfs2: fsid=MyCluster:gfs_fs.0: Glock dequeues delayed: 3
  [1657697.446964] gfs2: fsid=MyCluster:gfs_fs.0: telling LM to unmount
  [1657697.447011] dlm: gfs_fs: leaving the lockspace group...
  [1657697.447841] gfs2: fsid=MyCluster:gfs_fs.0: recover_prep ignored due to withdraw.
  

• A gfs2 filesystem withdrawal occurs then a kernel panic.

  [2487795.891294] gfs2: fsid=mycluster:data.0: fatal: assertion "!tr->tr_num_buf_new && !tr->tr_num_databuf_new" failed
     function = gfs2_log_flush, file = fs/gfs2/log.c, line = 1010
  [2487795.892318] CPU: 27 PID: 843646 Comm: kworker/27:1H Kdump: loaded Not tainted 4.18.0-553.56.1.el8_10.x86_64 #1
  [2487795.892327] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  [2487795.892334] Workqueue: gfs2-glock/pzmspw12_cl: glock_work_func [gfs2]
  [2487795.892401] Call Trace:
  [2487795.892412]  dump_stack+0x41/0x60
  [2487795.892424]  gfs2_log_flush+0xbe2/0xbf0 [gfs2]
  [2487795.892465]  ? sched_clock_cpu+0xc/0xb0
  [2487795.892480]  inode_go_sync+0x80/0x190 [gfs2]
  [2487795.892517]  do_xmote+0xe3/0x3f0 [gfs2]
  [2487795.892550]  ? __switch_to+0x10c/0x430
  [2487795.892557]  glock_work_func+0x5b/0x150 [gfs2]
  [2487795.892591]  process_one_work+0x1d3/0x390
  [2487795.892601]  ? process_one_work+0x390/0x390
  [2487795.892608]  worker_thread+0x30/0x390
  [2487795.892616]  ? process_one_work+0x390/0x390
  [2487795.892625]  kthread+0x134/0x150
  [2487795.892634]  ? set_kthread_struct+0x50/0x50
  [2487795.892642]  ret_from_fork+0x1f/0x40
  [2487795.892737] gfs2: fsid=mycluster:data.0: about to withdraw this file system
  [2487795.892754] gfs2: fsid=mycluster:data.0: GFS2:adding buf while withdrawn! 0x27d03b1
  [2487795.893070] gfs2: fsid=mycluster:data.0: GFS2:adding buf while withdrawn! 0x20be93d
  [2487795.893082] gfs2: fsid=mycluster:data.0: GFS2:adding buf while withdrawn! 0x27d03d1
  [2487802.662612] gfs2: fsid=mycluster:data.0: Requesting recovery of jid 0.
  [2487802.866861] gfs2: fsid=mycluster:data.0: No other mounters found. Trying to recover our own journal jid 0.
  [2487802.866882] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
  [2487802.867158] PGD 0 P4D 0
  [2487802.867297] Oops: 0000 [#1] SMP NOPTI
  [2487802.867401] CPU: 27 PID: 2972388 Comm: kworker/27:3 Kdump: loaded Not tainted 4.18.0-553.56.1.el8_10.x86_64 #1
  [2487802.867535] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  [2487802.867668] Workqueue: gfs_recovery gfs2_recover_func [gfs2]
  [2487802.867849] RIP: 0010:gfs2_recover_func+0x3e/0xf0 [gfs2]
  [2487802.867987] Code: 89 e5 41 57 41 56 41 55 41 54 41 52 53 48 89 fb 48 81 ec 60 01 00 00 4c 8b 6f 40 65 48 8b 04 25 28 00 00 00 48 89 45 c8 31 c0 <49> 8b 45 28 4c 8b b8 00 04 00 00 49 8b 87 90 00 00 00 a8 04 74 1b
  [2487802.868315] RSP: 0018:ffffb68e1856fcf0 EFLAGS: 00010246
  [2487802.868459] RAX: 0000000000000000 RBX: ffff8c0d4df64da8 RCX: dead000000000200
  [2487802.868603] RDX: 0000000000000001 RSI: ffff8c0d4df64db0 RDI: ffff8c0d4df64da8
  [2487802.868752] RBP: ffffb68e1856fe80 R08: 0000000000000000 R09: ffff8c0d821234b8
  [2487802.868931] R10: ffffb68e1856fe98 R11: 0000000000000000 R12: ffffd68dfc4fe400
  [2487802.869103] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8c0d4df64db0
  [2487802.869279] FS:  0000000000000000(0000) GS:ffff8c4b7e4c0000(0000) knlGS:0000000000000000
  [2487802.869469] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [2487802.869626] CR2: 0000000000000028 CR3: 000000337d610005 CR4: 00000000007706e0
  [2487802.869845] PKRU: 55555554
  [2487802.870031] Call Trace:
  [2487802.870240]  ? __die_body+0x1a/0x60
  [2487802.870413]  ? no_context+0x1ba/0x3f0
  [2487802.870586]  ? __bad_area_nosemaphore+0x157/0x180
  [2487802.870770]  ? vmw_stdu_plane_update_surface+0xc1/0xf0 [vmwgfx]
  [2487802.871010]  ? do_page_fault+0x37/0x12d
  [2487802.871203]  ? page_fault+0x1e/0x30
  [2487802.871412]  ? gfs2_recover_func+0x3e/0xf0 [gfs2]
  [2487802.871632]  ? __drm_atomic_state_free+0x65/0x70 [drm]
  [2487802.871871]  ? kfree+0xd3/0x250
  [2487802.872090]  ? drm_modeset_unlock+0x16/0x40 [drm]
  [2487802.872339]  ? next_online_pgdat+0x22/0x50
  [2487802.872532]  ? refresh_cpu_vm_stats+0x23d/0x280
  [2487802.872721]  ? update_load_avg+0x7e/0x710
  [2487802.872941]  ? newidle_balance+0x2b6/0x3b0
  [2487802.873134]  ? __switch_to_asm+0x43/0x80
  [2487802.873342]  ? __switch_to+0x10c/0x430
  [2487802.873528]  ? process_one_work+0x1d3/0x390
  [2487802.873706]  process_one_work+0x1d3/0x390
  [2487802.873906]  ? process_one_work+0x390/0x390
  [2487802.874108]  worker_thread+0x30/0x390
  [2487802.874293]  ? process_one_work+0x390/0x390
  [2487802.874504]  kthread+0x134/0x150
  [2487802.874687]  ? set_kthread_struct+0x50/0x50
  [2487802.874874]  ret_from_fork+0x1f/0x40
  [2487802.875069] Modules linked in: gfs2 dlm unix_diag nfnetlink ip6_udp_tunnel udp_tunnel vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc ext4 mbcache jbd2 sr_mod cdrom ata_generic vmwgfx sd_mod t10_pi drm_ttm_helper ttm sg drm_kms_helper syscopyarea ahci sysfillrect sysimgblt crc32c_intel ata_piix libahci drm libata vmxnet3 serio_raw vmw_pvscsi dm_mod fuse
  [2487802.876035] Red Hat flags: eBPF/event
  [2487802.876241] CR2: 0000000000000028