A cluster node unmounting a gfs2 filesystem triggered a kernel panic: kernel BUG at fs/inode.c:1680!

Solution In Progress - Updated -

Issue

  • A cluster node unmounting a gfs2 filesystem triggered a kernel panic:
[59311.891672] ------------[ cut here ]------------
[59311.891673] kernel BUG at fs/inode.c:1680!
[59311.891680] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[59311.891683] CPU: 43 PID: 1307791 Comm: umount Kdump: loaded Tainted: P           OE    --------- ---  5.14.0-70.13.1.el9_0.x86_64 #1
[59311.891686] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[59311.891687] RIP: 0010:iput+0x1aa/0x1d0
[59311.891693] Code: 5b 5d 41 5c e9 a7 f8 ff ff 48 8b 95 98 00 00 00 8b 4d 48 48 89 d0 85 c9 74 a1 48 83 bd e0 00 00 00 00 0f 85 3d ff ff ff eb 91 <0f> 0b c3 0f 0b e9 11 ff ff ff 48 89 ef e8 e4 fd ff ff 4c 89 e7 5b
[59311.891695] RSP: 0018:ffff96fbe03f7dd0 EFLAGS: 00010202
[59311.891698] RAX: ffff96fbe03f7df0 RBX: ffff96fbe03f7df0 RCX: 0000000080400017
[59311.891700] RDX: ffff96fbe03f7df0 RSI: ffff88a543d41ec0 RDI: ffff88a5c34d6bd8
[59311.891701] RBP: ffff88a5c34d6bd8 R08: ffff96fbe03f7df0 R09: 0000000000000000
[59311.891703] R10: 0000000000000001 R11: 0000000000000000 R12: dead000000000122
[59311.891704] R13: dead000000000100 R14: ffff88a5b3cea590 R15: 0000000000000000
[59311.891706] FS:  00007fea3c03c500(0000) GS:ffff88ad1fd40000(0000) knlGS:0000000000000000
[59311.891708] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59311.891710] CR2: 0000563d81749018 CR3: 00000008f3e2e002 CR4: 00000000003706e0
[59311.891711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[59311.891713] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[59311.891714] Call Trace:
[59311.891717]  gfs2_jindex_free+0x10b/0x150 [gfs2]
[59311.891732]  gfs2_put_super+0x156/0x1e0 [gfs2]
[59311.891745]  generic_shutdown_super+0x74/0x110
[59311.891748]  kill_block_super+0x21/0x50
[59311.891751]  deactivate_locked_super+0x36/0xa0
[59311.891754]  cleanup_mnt+0x131/0x190
[59311.891757]  task_work_run+0x5c/0x90
[59311.891760]  exit_to_user_mode_loop+0x152/0x160
[59311.891764]  exit_to_user_mode_prepare+0x89/0xd0
[59311.891768]  syscall_exit_to_user_mode+0x12/0x30
[59311.891771]  do_syscall_64+0x48/0x90
[59311.891774]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[59311.891776] RIP: 0033:0x7fea3c28aa6b
[59311.891778] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
[59311.891780] RSP: 002b:00007ffe50e91d18 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[59311.891782] RAX: 0000000000000000 RBX: 0000563d81749670 RCX: 00007fea3c28aa6b
[59311.891784] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000563d817496c0
[59311.891786] RBP: 0000563d81745390 R08: 0000000000000000 R09: 00007ffe50e90aa0
[59311.891787] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[59311.891791] Modules linked in: gfs2 dlm snd_seq_dummy snd_hrtimer snd_seq snd_timer snd_seq_device snd soundcore qrtr rfkill sunrpc emcpdm(POE) emcpgpx(POE) emcpmpx(POE) emcp(POE) intel_rapl_msr iTCO_wdt iTCO_vendor_support dcdbas intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass rapl intel_cstate intel_uncore pcspkr mxm_wmi lpc_ich mei_me mei ipmi_ssif mlx5_ib ib_uverbs ib_core ipmi_si acpi_power_meter xfs libcrc32c sr_mod cdrom sd_mod sg mgag200 drm_kms_helper mlx5_core syscopyarea sysfillrect sysimgblt fb_sys_fops cec qla2xxx crct10dif_pclmul crc32_pclmul drm crc32c_intel ixgbe ghash_clmulni_intel ahci megaraid_sas nvme_fc libahci nvme_fabrics libata nvme_core mlxfw mdio dca psample tls t10_pi scsi_transport_fc i2c_algo_bit pci_hyperv_intf wmi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse [last unloaded: emcpioc]
  • A cluster node has been encountering multiple kernel panics and below is a couple of snippets from the kernel panics:
[ 4466.020959] list_del corruption. next->prev should be ffffd5ba050bd508, but was ffffd5ba09c54388
[ 4466.020972] ------------[ cut here ]------------
[ 4466.020973] kernel BUG at lib/list_debug.c:54!
[ 4466.020979] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 4466.020982] CPU: 34 PID: 1602 Comm: kworker/34:1H Kdump: loaded Tainted: P    B      OE    --------- ---  5.14.0-70.13.1.el9_0.x86_64 #1
[ 4466.020985] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[ 4466.020987] Workqueue: glock_workqueue glock_work_func [gfs2]
[ 4466.021008] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47


[ 8257.691949] list_add double add: new=ffffddef467bfc08, prev=ffff9766bf96f798, next=ffffddef467bfc08.
[ 8257.691973] ------------[ cut here ]------------
[ 8257.691974] kernel BUG at lib/list_debug.c:29!
[ 8257.691984] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 8257.691990] CPU: 10 PID: 18732 Comm: kworker/u96:4 Kdump: loaded Tainted: P    B      OE    --------- ---  5.14.0-70.13.1.el9_0.x86_64 #1
[ 8257.691996] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.13.0 05/14/2021
[ 8257.691999] Workqueue: dlm_recv process_recv_sockets [dlm]
[ 8257.692029] RIP: 0010:__list_add_valid.cold+0x26/0x3f

Environment

  • Red Hat Enterprise Linux Server 9 (with the High Availability Add On and Resilient Storage Add Ons)
  • A Global Filesystem 2(gfs2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content