RHEL 9: Kernel panic - not syncing: Hard LOCKUP with evict_folios

Issue

• Kernel panic with logs:

[1473947.629962] NMI watchdog: Watchdog detected hard LOCKUP on cpu 3
[1473947.629972] Modules linked in: isofs cdrom raid0 tls rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vfat fat vmw_balloon intel_rapl_msr intel_rapl_common pcspkr vmw_vmci i2c_piix4 joydev auth_rpcgss sch_fq tcp_bbr sunrpc ext4 mbcache jbd2 ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper sd_mod syscopyarea sysfillrect sg sysimgblt fb_sys_fops ahci crct10dif_pclmul crc32_pclmul ata_piix libahci crc32c_intel nvme drm libata ghash_clmulni_intel nvme_core vmxnet3 nvme_common vmw_pvscsi t10_pi serio_raw fuse
[1473947.630043] CPU: 3 PID: 355999 Comm: sas Kdump: loaded Not tainted 5.14.0-427.37.1.el9_4.x86_64 #1
[1473947.630049] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[1473947.630051] RIP: 0010:native_queued_spin_lock_slowpath+0x32/0x2b0
[1473947.630064] Code: 54 55 48 89 fd 53 66 90 ba 01 00 00 00 8b 45 00 85 c0 75 14 f0 0f b1 55 00 85 c0 75 f0 5b 5d 41 5c 41 5d e9 90 12 00 00 f3 90 <eb> e1 81 fe 00 01 00 00 74 52 40 30 f6 85 f6 75 75 f0 0f ba 6d 00
[1473947.630066] RSP: 0018:ffffbe94c11f7798 EFLAGS: 00000002
[1473947.630068] RAX: 0000000000000001 RBX: 00000000043ad5ab RCX: 0000000000000000
[1473947.630070] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9c48c2271050
[1473947.630071] RBP: ffff9c48c2271050 R08: 0000000000000000 R09: 0000000004349c73
[1473947.630072] R10: 00000000043ad5ab R11: 0000000000000000 R12: ffffbe94c11f7818
[1473947.630073] R13: ffff9c48c2271000 R14: ffffbe94c11f7ac8 R15: 000000000000001e
[1473947.630074] FS:  00007f4ae29ff640(0000) GS:ffff9cc04bac0000(0000) knlGS:0000000000000000
[1473947.630075] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1473947.630076] CR2: 00007f2758a04274 CR3: 0000000331bb2000 CR4: 00000000003506e0
[1473947.630105] Call Trace:
[1473947.630112]  <NMI>
[1473947.630114]  ? srso_return_thunk+0x5/0x5f
[1473947.630118]  ? show_trace_log_lvl+0x26e/0x2df
[1473947.630130]  ? show_trace_log_lvl+0x26e/0x2df
[1473947.630133]  ? _raw_spin_lock_irq+0x26/0x30
[1473947.630136]  ? watchdog_overflow_callback+0xd0/0x130
[1473947.630142]  ? __perf_event_overflow+0x105/0x1e0
[1473947.630149]  ? x86_pmu_handle_irq+0x147/0x1b0
[1473947.630162]  ? amd_pmu_handle_irq+0x42/0xb0
[1473947.630164]  ? perf_event_nmi_handler+0x28/0x50
[1473947.630167]  ? nmi_handle+0x5e/0x120
[1473947.630171]  ? default_do_nmi+0x40/0x130
[1473947.630177]  ? exc_nmi+0x111/0x140
[1473947.630179]  ? end_repeat_nmi+0x16/0x67
[1473947.630189]  ? native_queued_spin_lock_slowpath+0x32/0x2b0
[1473947.630190]  ? native_queued_spin_lock_slowpath+0x32/0x2b0
[1473947.630192]  ? native_queued_spin_lock_slowpath+0x32/0x2b0
[1473947.630194]  </NMI>
[1473947.630195]  <TASK>
[1473947.630196]  _raw_spin_lock_irq+0x26/0x30
[1473947.630198]  evict_folios+0x80/0x620
[1473947.630205]  ? list_lru_count_one+0x63/0x90
[1473947.630211]  ? srso_return_thunk+0x5/0x5f
[1473947.630212]  ? do_shrink_slab+0x3a/0x2f0
[1473947.630215]  try_to_shrink_lruvec+0x186/0x2a0
[1473947.630220]  shrink_one+0x111/0x1e0
[1473947.630223]  shrink_many+0x9f/0x270
[1473947.630227]  shrink_node+0x406/0x4a0
[1473947.630230]  shrink_zones.constprop.0+0x88/0x280
[1473947.630233]  do_try_to_free_pages+0x92/0x2d0
[1473947.630235]  try_to_free_pages+0xd8/0x200
[1473947.630239]  __alloc_pages_slowpath.constprop.0+0x344/0x960
[1473947.630244]  ? srso_return_thunk+0x5/0x5f
[1473947.630246]  ? get_page_from_freelist+0xe8/0x530
[1473947.630251]  __alloc_pages+0x21d/0x250
[1473947.630254]  folio_alloc+0x17/0x50
[1473947.630259]  __filemap_get_folio+0x1cd/0x330
[1473947.630269]  nfs_write_begin+0x52/0x2c0 [nfs]
[1473947.630324]  generic_perform_write+0xbc/0x200
[1473947.630327]  ? reweight_entity+0x14d/0x160
[1473947.630334]  ? srso_return_thunk+0x5/0x5f
[1473947.630335]  ? nfs_ctx_key_to_expire+0xd4/0x120 [nfs]
[1473947.630352]  nfs_file_write+0x18d/0x280 [nfs]
[1473947.630367]  vfs_write+0x2ce/0x410
[1473947.630373]  ksys_write+0x5f/0xe0
[1473947.630375]  do_syscall_64+0x5c/0x90
[1473947.630379]  ? srso_return_thunk+0x5/0x5f
[1473947.630381]  ? __irq_exit_rcu+0x46/0xc0
[1473947.630386]  ? srso_return_thunk+0x5/0x5f
[1473947.630387]  ? sysvec_apic_timer_interrupt+0x3c/0x90
[1473947.630389]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[1473947.630392] RIP: 0033:0x7f4b1bafda8f
[1473947.630434] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 c9 85 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 1c 86 f8 ff 48
[1473947.630435] RSP: 002b:00007f4ae29ed7a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[1473947.630437] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4b1bafda8f
[1473947.630438] RDX: 0000000000010000 RSI: 00007f4afcecd440 RDI: 0000000000000022
[1473947.630439] RBP: 00007f4ae29ed8d8 R08: 0000000000000000 R09: 0000000000000001
[1473947.630440] R10: 00007f4b16b41bad R11: 0000000000000293 R12: 0000000000010000
[1473947.630440] R13: 00007f4afcecd440 R14: 00007f4b16fb4600 R15: 00007f4b1adc6bd8
[1473947.630444]  </TASK>
[1473947.630448] Kernel panic - not syncing: Hard LOCKUP