RHEL8.6: Soft lockup occurred at nfs_do_recoalesce

Issue

• Soft lockup occurred with below logs:

May 24 13:58:50 rh86-01 kernel: watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [bash:11610]
May 24 13:58:50 rh86-01 kernel: Modules linked in: nfs_layout_nfsv41_files rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr i2c_piix4 vmw_vmci binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic vmwgfx sd_mod t10_pi sg ttm drm_kms_helper ahci syscopyarea sysfillrect sysimgblt libahci ata_piix crc32c_intel fb_sys_fops serio_raw drm libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse
May 24 13:58:50 rh86-01 kernel: CPU: 9 PID: 11610 Comm: bash Kdump: loaded Not tainted 4.18.0-372.9.1.el8.x86_64 #1
May 24 13:58:50 rh86-01 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
May 24 13:58:50 rh86-01 kernel: RIP: 0010:nfs_do_recoalesce+0x74/0xf0 [nfs]
May 24 13:58:50 rh86-01 kernel: Code: 08 48 8b 03 48 39 c3 74 21 48 8b 0b 48 8b 04 24 48 8b 53 08 48 89 0c 24 4c 89 61 08 48 89 02 48 89 50 08 48 89 1b 48 89 5b 08 <48> 8b 04 24 49 39 c4 74 17 48 8b 34 24 48 89 ef e8 f7 fa ff ff 85
May 24 13:58:50 rh86-01 kernel: RSP: 0018:ffff9ad8c65a7c88 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
May 24 13:58:50 rh86-01 kernel: RAX: ffff9ad8c65a7d40 RBX: ffff9ad8c65a7d40 RCX: 000000000000000c
May 24 13:58:50 rh86-01 kernel: RDX: 000000000000000b RSI: 1a9806e3341a8f31 RDI: ffff9ad8c65a7ce8
May 24 13:58:50 rh86-01 kernel: RBP: ffff9ad8c65a7ce8 R08: 0000000080000000 R09: 0000000000000001
May 24 13:58:50 rh86-01 kernel: R10: ffff8ef4e0952898 R11: 000000000005881b R12: ffff9ad8c65a7c88
May 24 13:58:50 rh86-01 kernel: R13: ffff9ad8c65a7d40 R14: 000000010ebb0c81 R15: ffff8ef6707780b0
May 24 13:58:50 rh86-01 kernel: FS:  00007fe6358e1740(0000) GS:ffff8f04f7c40000(0000) knlGS:0000000000000000
May 24 13:58:50 rh86-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 24 13:58:50 rh86-01 kernel: CR2: 000055c19084dc7c CR3: 00000003771c0001 CR4: 00000000003706e0
May 24 13:58:50 rh86-01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 24 13:58:50 rh86-01 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 24 13:58:50 rh86-01 kernel: Call Trace:
May 24 13:58:50 rh86-01 kernel: nfs_pageio_complete+0x6c/0x120 [nfs]
May 24 13:58:50 rh86-01 kernel: nfs_writepages+0xc6/0x1e0 [nfs]
May 24 13:58:50 rh86-01 kernel: do_writepages+0xc2/0x1c0
May 24 13:58:50 rh86-01 kernel: ? unx_destroy+0x10/0x10 [sunrpc]
May 24 13:58:50 rh86-01 kernel: ? nfs_ctx_key_to_expire+0x67/0xf0 [nfs]
May 24 13:58:50 rh86-01 kernel: __filemap_fdatawrite_range+0xc7/0x100
May 24 13:58:50 rh86-01 kernel: filemap_write_and_wait+0x2c/0x70
May 24 13:58:50 rh86-01 kernel: nfs_wb_all+0x1a/0x120 [nfs]
May 24 13:58:50 rh86-01 kernel: nfs4_file_flush+0x6b/0xa0 [nfsv4]
May 24 13:58:50 rh86-01 kernel: filp_close+0x31/0x60
May 24 13:58:50 rh86-01 kernel: __x64_sys_close+0x1e/0x50
May 24 13:58:50 rh86-01 kernel: do_syscall_64+0x5b/0x1a0
May 24 13:58:50 rh86-01 kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
May 24 13:58:50 rh86-01 kernel: RIP: 0033:0x7fe634ff7c18
May 24 13:58:50 rh86-01 kernel: Code: e6 01 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 85 39 2a 00 8b 00 85 c0 75 17 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 40 c3 0f 1f 80 00 00 00 00 53 89 fb 48 83 ec
May 24 13:58:50 rh86-01 kernel: RSP: 002b:00007ffe569af948 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
May 24 13:58:50 rh86-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000072 RCX: 00007fe634ff7c18
May 24 13:58:50 rh86-01 kernel: RDX: 000055c19096a790 RSI: 0000000000000005 RDI: 0000000000000004
May 24 13:58:50 rh86-01 kernel: RBP: 000055c19096a090 R08: 000055c19084401c R09: 0000000000000004
May 24 13:58:50 rh86-01 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055c1909ca630
May 24 13:58:50 rh86-01 kernel: R13: 0000000000000000 R14: 000055c1909f7710 R15: 000055c1909ca630