NMI Watchdog detected Hard Lockup in vmballoon_enqueue_page_list()

Issue

• Kernel panic with the following call traces:

[1763249.651765] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
[1763249.651872] Modules linked in: tls rpcrdma rdma_cm iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vfat fat intel_rapl_msr intel_rapl_common vmw_balloon pcspkr vmw_vmci i2c_piix4 joydev nfsd nfs_acl lockd auth_rpcgss grace sunrpc xfs libcrc32c vmwgfx sr_mod cdrom drm_ttm_helper ttm sd_mod ata_generic t10_pi drm_kms_helper sg ahci crct10dif_pclmul libahci ata_piix crc32_pclmul crc32c_intel drm ghash_clmulni_intel libata vmw_pvscsi vmxnet3 serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
[1763249.652201] CPU: 0 PID: 3051883 Comm: kworker/0:3 Kdump: loaded Tainted: GL ----------  5.14.0-503.35.1.el9_5.x86_64 #1
[1763249.652208] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[1763249.652214] Workqueue: events_freezable vmballoon_work [vmw_balloon]
[1763249.652278] RIP: 0010:vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.652284] Code: a7 b3 f9 4c 8d 6f 58 4c 89 ef e8 30 8c b3 f9 48 89 c6 48 8b 45 00 48 8d 48 f8 48 39 c5 74 27 48 8d 41 30 48 8d 91 30 80 00 00 <81> 20 ff fe ff ff 48 83 c0 40 48 39 c2 75 f1 48 8b 41 08 48 8d 48
[1763249.652286] RSP: 0018:ffffbc8602ba3d88 EFLAGS: 00000006
[1763249.652290] RAX: fffff30df7239030 RBX: ffffffffc09e8780 RCX: fffff30df7238000
[1763249.652422] RDX: fffff30df7240030 RSI: 0000000000000202 RDI: ffffffffc09e87d8
[1763249.652423] RBP: ffffbc8602ba3dd0 R08: ffffffffc09e8780 R09: ffffbc8602ba3d68
[1763249.652424] R10: 00000000001089b9 R11: 0000000000000001 R12: ffffbc8602ba3e04
[1763249.652426] R13: ffffffffc09e87d8 R14: fffff30df65f0008 R15: 0000000000000009
[1763249.652427] FS:  0000000000000000(0000) GS:ffff9c582fc00000(0000) knlGS:0000000000000000
[1763249.652429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1763249.652439] CR2: 00007f0858024ff8 CR3: 000000025c326000 CR4: 0000000000350ef0
[1763249.652456] Call Trace:
[1763249.652462]  <NMI>
[1763249.652464]  ? srso_return_thunk+0x5/0x5f
[1763249.652769]  ? show_trace_log_lvl+0x26e/0x2df
[1763249.653058]  ? show_trace_log_lvl+0x26e/0x2df
[1763249.653071]  ? vmballoon_inflate+0x1b6/0x400 [vmw_balloon]
[1763249.653081]  ? watchdog_overflow_callback.cold+0x1e/0x70
[1763249.653093]  ? __perf_event_overflow+0x105/0x1e0
[1763249.653214]  ? x86_pmu_handle_irq+0x14a/0x1d0
[1763249.653257]  ? amd_pmu_handle_irq+0x42/0xb0
[1763249.653261]  ? perf_event_nmi_handler+0x28/0x50
[1763249.653266]  ? nmi_handle+0x5e/0x120
[1763249.653278]  ? default_do_nmi+0x40/0x130
[1763249.653508]  ? exc_nmi+0x100/0x180
[1763249.653514]  ? end_repeat_nmi+0xf/0x60
[1763249.653537]  ? vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.653546]  ? vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.653556]  ? vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.653575]  </NMI>
[1763249.653576]  <TASK>
[1763249.653580]  vmballoon_inflate+0x1b6/0x400 [vmw_balloon]
[1763249.653600]  vmballoon_work+0xfe/0x195 [vmw_balloon]
[1763249.653607]  process_one_work+0x197/0x380
[1763249.653772]  worker_thread+0x2fe/0x410
[1763249.653778]  ? __pfx_worker_thread+0x10/0x10
[1763249.653782]  kthread+0xe0/0x100
[1763249.653785]  ? __pfx_kthread+0x10/0x10
[1763249.653791]  ret_from_fork+0x2c/0x50
[1763249.653892]  </TASK>
[1763249.653900] Kernel panic - not syncing: Hard LOCKUP
[1763249.653902] CPU: 0 PID: 3051883 Comm: kworker/0:3 Kdump: loaded Tainted: GL ----------  5.14.0-503.35.1.el9_5.x86_64 #1
[1763249.653904] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[1763249.653906] Workqueue: events_freezable vmballoon_work [vmw_balloon]
[1763249.653911] Call Trace:
[1763249.653922]  <NMI>
[1763249.653926]  dump_stack_lvl+0x34/0x48
[1763249.653937]  panic+0x107/0x2bb
[1763249.653961]  nmi_panic.cold+0xc/0xc
[1763249.653965]  watchdog_overflow_callback.cold+0x5c/0x70
[1763249.653977]  __perf_event_overflow+0x105/0x1e0
[1763249.653990]  x86_pmu_handle_irq+0x14a/0x1d0
[1763249.654038]  amd_pmu_handle_irq+0x42/0xb0
[1763249.654043]  perf_event_nmi_handler+0x28/0x50
[1763249.654048]  nmi_handle+0x5e/0x120
[1763249.654057]  default_do_nmi+0x40/0x130
[1763249.654069]  exc_nmi+0x100/0x180
[1763249.654074]  end_repeat_nmi+0xf/0x60
[1763249.654077] RIP: 0010:vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.654082] Code: a7 b3 f9 4c 8d 6f 58 4c 89 ef e8 30 8c b3 f9 48 89 c6 48 8b 45 00 48 8d 48 f8 48 39 c5 74 27 48 8d 41 30 48 8d 91 30 80 00 00 <81> 20 ff fe ff ff 48 83 c0 40 48 39 c2 75 f1 48 8b 41 08 48 8d 48
[1763249.654084] RSP: 0018:ffffbc8602ba3d88 EFLAGS: 00000006
[1763249.654086] RAX: fffff30df7239030 RBX: ffffffffc09e8780 RCX: fffff30df7238000
[1763249.654087] RDX: fffff30df7240030 RSI: 0000000000000202 RDI: ffffffffc09e87d8
[1763249.654089] RBP: ffffbc8602ba3dd0 R08: ffffffffc09e8780 R09: ffffbc8602ba3d68
[1763249.654090] R10: 00000000001089b9 R11: 0000000000000001 R12: ffffbc8602ba3e04
[1763249.654092] R13: ffffffffc09e87d8 R14: fffff30df65f0008 R15: 0000000000000009
[1763249.654107]  ? vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.654114]  ? vmballoon_enqueue_page_list+0x5b/0xe0 [vmw_balloon]
[1763249.654121]  </NMI>
[1763249.654121]  <TASK>
[1763249.654124]  vmballoon_inflate+0x1b6/0x400 [vmw_balloon]
[1763249.654144]  vmballoon_work+0xfe/0x195 [vmw_balloon]
[1763249.654152]  process_one_work+0x197/0x380
[1763249.654158]  worker_thread+0x2fe/0x410
[1763249.654165]  ? __pfx_worker_thread+0x10/0x10
[1763249.654169]  kthread+0xe0/0x100
[1763249.654172]  ? __pfx_kthread+0x10/0x10
[1763249.654177]  ret_from_fork+0x2c/0x50
[1763249.654188]  </TASK>