RHEL9: softlockup on _raw_spin_unlock_irqrestore calling from fq_flush_timeout()

Issue

• Soft lockup occurred with logs:

[  183.022629] nvme nvme1: rescanning namespaces.
[39940.529629] watchdog: BUG: soft lockup - CPU#98 stuck for 24s! [swapper/98:0]
[39940.529662] Modules linked in: dm_mod nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_rej
ect_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c 
nfnetlink qrtr sunrpc vfat fat intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm irqbypass rapl pcspkr ipm
i_ssif joydev acpi_ipmi ses enclosure i2c_designware_platform k10temp i2c_piix4 ipmi_si i2c_designware_core acpi_cpufreq ext4
 mbcache jbd2 sd_mod sg crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ast i2c_algo_bit drm_vram_helper drm_k
ms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm_ttm_helper nvme ahci ttm libahci smartpqi(OE) nvme_core drm l
ibata t10_pi ccp scsi_transport_sas ax88179_178a usbnet mii ipmi_devintf ipmi_msghandler fuse
[39940.529707] CPU: 98 PID: 0 Comm: swapper/98 Kdump: loaded Tainted: G           OE    --------- ---  5.14.0-70.22.1.el9_0.x
86_64 #1
[39940.529710] Hardware name: A1 A2/B2, BIOS 04.02.23 06/25/2023
[39940.529711] RIP: 0010:_raw_spin_unlock_irqrestore+0x1c/0x30
[39940.529716] Code: 00 00 eb f5 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 c6 07 00 0f 1f 40 00 f7 c6 00 02 00 00 74 01 fb
 65 ff 0d c4 a2 58 7a <74> 01 c3 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[39940.529717] RSP: 0018:ff46f2999b050e98 EFLAGS: 00000202
[39940.529719] RAX: ff78f249819da738 RBX: ff18cc0c27a6af28 RCX: 0000000000000000
[39940.529719] RDX: ff78f249819db7a0 RSI: 0000000000000282 RDI: ff78f249819db7a0
[39940.529720] RBP: ff78f249819d9798 R08: 0000000000000000 R09: 0000000000000000
[39940.529721] R10: 00000000000000fb R11: 0000000000000100 R12: 00000000000000fb
[39940.529721] R13: ff18cc0c27a6a808 R14: 0000000000000282 R15: ff78f249819db7a0
[39940.529722] FS:  0000000000000000(0000) GS:ff18cca9c9180000(0000) knlGS:0000000000000000
[39940.529723] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39940.529723] CR2: 00007f1fa37dc004 CR3: 000000514e442001 CR4: 0000000000771ee0
[39940.529724] PKRU: 55555554
[39940.529725] Call Trace:
[39940.529726]  <IRQ>
[39940.529727]  fq_flush_timeout+0x6d/0xa0
[39940.529733]  ? fq_ring_free+0xb0/0xb0
[39940.529735]  ? fq_ring_free+0xb0/0xb0
[39940.529736]  call_timer_fn+0x26/0x130
[39940.529739]  __run_timers.part.0+0x1c6/0x250
[39940.529740]  ? __hrtimer_run_queues+0x139/0x2c0
[39940.529742]  ? recalibrate_cpu_khz+0x10/0x10
[39940.529744]  ? ktime_get+0x38/0x90
[39940.529746]  run_timer_softirq+0x26/0x50
[39940.529747]  __do_softirq+0xca/0x2a8
[39940.529749]  __irq_exit_rcu+0xad/0xe0
[39940.529751]  sysvec_apic_timer_interrupt+0x72/0x90
[39940.529753]  </IRQ>
[39940.529754]  asm_sysvec_apic_timer_interrupt+0x12/0x20