A kernel panic is occurring on a cluster node with gfs2 filesystems mounted: `kernel BUG at fs/gfs2/lops.c:135!`

Issue

• There are quota related assertions that occur:

Jun 23 07:19:11 node42 kernel: gfs2: fsid=Cluster-PROD:data3.2: warning: assertion "!ip->i_qadata->qa_qd_num" failed at function = gfs2_quota_hold, file = fs/gfs2/quota.c, line = 572
Jun 23 08:37:11 node42 kernel: gfs2: fsid=Cluster-PROD:data3.2: warning: assertion "ip->i_qadata->qa_ref == 0" failed at function = gfs2_evict_inode, file = fs/gfs2/super.c, line = 1660

• A kernel panic is occurring on a cluster node with gfs2 filesystems mounted: kernel BUG at fs/gfs2/lops.c:135!

[553186.917374] ------------[ cut here ]------------
[553186.919298] kernel BUG at fs/gfs2/lops.c:135!
[553186.920501] invalid opcode: 0000 [#1] SMP
[553186.921708] Modules linked in: ip6table_filter ip6_tables iptable_filter vsock_diag unix_diag af_packet_diag netlink_diag falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14810(E) gfs2 dlm falcon_lsm_pinned_14203(E) tcp_diag udp_diag inet_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common drm crc32c_intel
[553186.931342]  libata nfit serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
[553186.936019] CPU: 0 PID: 56110 Comm: kworker/0:98H Kdump: loaded Tainted: P            E  ------------   3.10.0-1160.88.1.el7.x86_64 #1
[553186.938618] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[553186.941357] Workqueue: glock_workqueue glock_work_func [gfs2]
[553186.944128] task: ffff9ff6fac03180 ti: ffff9ff761d78000 task.ti: ffff9ff761d78000
[553186.946970] RIP: 0010:[<ffffffffc0aada77>]  [<ffffffffc0aada77>] gfs2_log_write+0x197/0x1a0 [gfs2]
[553186.949947] RSP: 0018:ffff9ff761d7bb20  EFLAGS: 00010202
[553186.952921] RAX: ffff9ff46511b300 RBX: ffffefdd8dfbfc00 RCX: ffff9ff465073210
[553186.955986] RDX: 00000000000192e4 RSI: 0000000000020000 RDI: ffff9ff465073200
[553186.959080] RBP: ffff9ff761d7bb50 R08: 0000000000000000 R09: 0000000000000000
[553186.962207] R10: 0000000000000000 R11: ffff9ff73eff002c R12: 0000000000001000
[553186.965370] R13: 0000000000000000 R14: 000000000003950e R15: ffff9ff78d97b000
[553186.968555] FS:  0000000000000000(0000) GS:ffff9ff7ed600000(0000) knlGS:0000000000000000
[553186.971822] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[553186.975117] CR2: 00007fb133e1a3b0 CR3: 00000003c22f4000 CR4: 00000000001607f0
[553186.978519] Call Trace:
[553186.981915]  [<ffffffffc0aae0a7>] gfs2_log_write_page+0x17/0x20 [gfs2]
[553186.985410]  [<ffffffffc0aab14b>] log_write_header+0x13b/0x440 [gfs2]
[553186.989152]  [<ffffffffc0aaafe8>] ? log_flush_wait+0xa8/0xd0 [gfs2]
[553186.992716]  [<ffffffffc0aabab6>] gfs2_log_flush+0x266/0x570 [gfs2]
[553186.996299]  [<ffffffffc0aa9b7e>] gfs2_ail_empty_gl+0xae/0x1b0 [gfs2]
[553186.999914]  [<ffffffffc0aaa1a1>] ? inode_go_sync+0xc1/0x190 [gfs2]
[553187.003560]  [<ffffffffc0aaa1a1>] inode_go_sync+0xc1/0x190 [gfs2]
[553187.007252]  [<ffffffffc0aa84a8>] do_xmote+0x148/0x460 [gfs2]
[553187.010952]  [<ffffffffc0aa8900>] run_queue+0x140/0x270 [gfs2]
[553187.014679]  [<ffffffffc0aa8a94>] glock_work_func+0x64/0x140 [gfs2]
[553187.018498]  [<ffffffff90ac32ef>] process_one_work+0x17f/0x440
[553187.022278]  [<ffffffff90ac4436>] worker_thread+0x126/0x3c0
[553187.026079]  [<ffffffff90ac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[553187.029939]  [<ffffffff90acb621>] kthread+0xd1/0xe0
[553187.033804]  [<ffffffff90acb550>] ? insert_kthread_work+0x40/0x40
[553187.037740]  [<ffffffff911c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[553187.041692]  [<ffffffff90acb550>] ? insert_kthread_work+0x40/0x40
[553187.045691] Code: 5c 0f 85 f5 fe ff ff 41 c7 87 68 08 00 00 00 00 00 00 49 c7 87 70 08 00 00 01 00 00 00 e9 da fe ff ff 41 3b 97 f0 07 00 00 74 c9 <0f> 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 f8 48 05 60
[553187.054720] RIP  [<ffffffffc0aada77>] gfs2_log_write+0x197/0x1a0 [gfs2]
[553187.059094]  RSP <ffff9ff761d7bb20>