A cluster node unmounting a gfs2 filesystem triggered a kernel panic: kernel BUG at fs/inode.c:1680!
Issue
- A cluster node unmounting a gfs2 filesystem triggered a kernel panic:
[59311.891672] ------------[ cut here ]------------
[59311.891673] kernel BUG at fs/inode.c:1680!
[59311.891680] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[59311.891683] CPU: 43 PID: 1307791 Comm: umount Kdump: loaded Tainted: P OE --------- --- 5.14.0-70.13.1.el9_0.x86_64 #1
[59311.891686] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[59311.891687] RIP: 0010:iput+0x1aa/0x1d0
[59311.891693] Code: 5b 5d 41 5c e9 a7 f8 ff ff 48 8b 95 98 00 00 00 8b 4d 48 48 89 d0 85 c9 74 a1 48 83 bd e0 00 00 00 00 0f 85 3d ff ff ff eb 91 <0f> 0b c3 0f 0b e9 11 ff ff ff 48 89 ef e8 e4 fd ff ff 4c 89 e7 5b
[59311.891695] RSP: 0018:ffff96fbe03f7dd0 EFLAGS: 00010202
[59311.891698] RAX: ffff96fbe03f7df0 RBX: ffff96fbe03f7df0 RCX: 0000000080400017
[59311.891700] RDX: ffff96fbe03f7df0 RSI: ffff88a543d41ec0 RDI: ffff88a5c34d6bd8
[59311.891701] RBP: ffff88a5c34d6bd8 R08: ffff96fbe03f7df0 R09: 0000000000000000
[59311.891703] R10: 0000000000000001 R11: 0000000000000000 R12: dead000000000122
[59311.891704] R13: dead000000000100 R14: ffff88a5b3cea590 R15: 0000000000000000
[59311.891706] FS: 00007fea3c03c500(0000) GS:ffff88ad1fd40000(0000) knlGS:0000000000000000
[59311.891708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59311.891710] CR2: 0000563d81749018 CR3: 00000008f3e2e002 CR4: 00000000003706e0
[59311.891711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[59311.891713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[59311.891714] Call Trace:
[59311.891717] gfs2_jindex_free+0x10b/0x150 [gfs2]
[59311.891732] gfs2_put_super+0x156/0x1e0 [gfs2]
[59311.891745] generic_shutdown_super+0x74/0x110
[59311.891748] kill_block_super+0x21/0x50
[59311.891751] deactivate_locked_super+0x36/0xa0
[59311.891754] cleanup_mnt+0x131/0x190
[59311.891757] task_work_run+0x5c/0x90
[59311.891760] exit_to_user_mode_loop+0x152/0x160
[59311.891764] exit_to_user_mode_prepare+0x89/0xd0
[59311.891768] syscall_exit_to_user_mode+0x12/0x30
[59311.891771] do_syscall_64+0x48/0x90
[59311.891774] entry_SYSCALL_64_after_hwframe+0x44/0xae
[59311.891776] RIP: 0033:0x7fea3c28aa6b
[59311.891778] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
[59311.891780] RSP: 002b:00007ffe50e91d18 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[59311.891782] RAX: 0000000000000000 RBX: 0000563d81749670 RCX: 00007fea3c28aa6b
[59311.891784] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000563d817496c0
[59311.891786] RBP: 0000563d81745390 R08: 0000000000000000 R09: 00007ffe50e90aa0
[59311.891787] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[59311.891791] Modules linked in: gfs2 dlm snd_seq_dummy snd_hrtimer snd_seq snd_timer snd_seq_device snd soundcore qrtr rfkill sunrpc emcpdm(POE) emcpgpx(POE) emcpmpx(POE) emcp(POE) intel_rapl_msr iTCO_wdt iTCO_vendor_support dcdbas intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass rapl intel_cstate intel_uncore pcspkr mxm_wmi lpc_ich mei_me mei ipmi_ssif mlx5_ib ib_uverbs ib_core ipmi_si acpi_power_meter xfs libcrc32c sr_mod cdrom sd_mod sg mgag200 drm_kms_helper mlx5_core syscopyarea sysfillrect sysimgblt fb_sys_fops cec qla2xxx crct10dif_pclmul crc32_pclmul drm crc32c_intel ixgbe ghash_clmulni_intel ahci megaraid_sas nvme_fc libahci nvme_fabrics libata nvme_core mlxfw mdio dca psample tls t10_pi scsi_transport_fc i2c_algo_bit pci_hyperv_intf wmi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse [last unloaded: emcpioc]
- A cluster node has been encountering multiple kernel panics and below is a couple of snippets from the kernel panics:
[ 4466.020959] list_del corruption. next->prev should be ffffd5ba050bd508, but was ffffd5ba09c54388
[ 4466.020972] ------------[ cut here ]------------
[ 4466.020973] kernel BUG at lib/list_debug.c:54!
[ 4466.020979] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 4466.020982] CPU: 34 PID: 1602 Comm: kworker/34:1H Kdump: loaded Tainted: P B OE --------- --- 5.14.0-70.13.1.el9_0.x86_64 #1
[ 4466.020985] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[ 4466.020987] Workqueue: glock_workqueue glock_work_func [gfs2]
[ 4466.021008] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47
[ 8257.691949] list_add double add: new=ffffddef467bfc08, prev=ffff9766bf96f798, next=ffffddef467bfc08.
[ 8257.691973] ------------[ cut here ]------------
[ 8257.691974] kernel BUG at lib/list_debug.c:29!
[ 8257.691984] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 8257.691990] CPU: 10 PID: 18732 Comm: kworker/u96:4 Kdump: loaded Tainted: P B OE --------- --- 5.14.0-70.13.1.el9_0.x86_64 #1
[ 8257.691996] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[ 8257.691999] Workqueue: dlm_recv process_recv_sockets [dlm]
[ 8257.692029] RIP: 0010:__list_add_valid.cold+0x26/0x3f
Environment
- Red Hat Enterprise Linux Server 9 (with the High Availability Add On and Resilient Storage Add Ons)
- A Global Filesystem 2(
gfs2
)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.