RHEL 8: Soft lockup occurred on __rpc_execute()

Issue

• Soft lockup ocfurred with logs:

[21650163.621101] watchdog: BUG: soft lockup - CPU#51 stuck for 23s! [kubelet:39688]
[21650163.621104] Modules linked in: bmhook(OE) ScvTOS50(OE) ScvTOS50Hook(OE) tmhook(OE) sha512_ssse3 sha512_generic udp_diag tcp_diag inet_diag ip_set_hash_ip ip_set_hash_net veth xt_recent xt_statistic xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_MASQUERADE ipt_MASQUERADE nft_counter nft_compat nft_chain_nat nf_nat nf_tables cfg80211 rfkill vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 br_netfilter bridge stp llc overlay ipip tunnel4 ip_tunnel xt_sctp xt_multiport xt_mark xt_ipvs ip_vs xt_conntrack xt_comment xt_addrtype nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_set binfmt_misc ipt_rpfilter ipt_REJECT nf_reject_ipv4 ip6_tables ip_set nfnetlink ip_tables xfs libcrc32c sr_mod cdrom
[21650163.621137]  ata_generic vmwgfx sd_mod t10_pi sg drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: bmhook]
[21650163.621148] Features: eBPF/xdp
[21650163.621151] CPU: 51 PID: 39688 Comm: kubelet Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-305.el8.x86_64 #1
[21650163.621152] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[21650163.621178] RIP: 0010:call_connect+0x0/0x90 [sunrpc]
[21650163.621180] Code: 48 c7 43 20 70 70 9a c0 48 89 df be 3e 00 00 00 5b 5d 41 5c 41 5d 41 5e e9 5d ed 00 00 0f 0b 0f 0b 66 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 53 48 8b 47 30 48 89 fb a8 08 74 52 48 8b 87 d0 00
[21650163.621181] RSP: 0018:ffffa202db95bbb0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[21650163.621182] RAX: 000000000000001d RBX: ffff8a2a5b994800 RCX: 0000000000000001
[21650163.621183] RDX: 000000060a6a603c RSI: ffff8a2a5b994800 RDI: ffff8a2a5b994800
[21650163.621183] RBP: ffff8a28915c91e8 R08: 000000000aaf89aa R09: 000000000aaf89aa
[21650163.621184] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000000
[21650163.621184] R13: 0000000000000000 R14: ffffffffc09b4b20 R15: ffffffffc09a7200
[21650163.621185] FS:  00007f08d8ff9700(0000) GS:ffff8a2b5eac0000(0000) knlGS:0000000000000000
[21650163.621186] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21650163.621186] CR2: 000000c00577b010 CR3: 0000001126e10001 CR4: 00000000007706e0
[21650163.621214] PKRU: 55555554
[21650163.621214] Call Trace:
[21650163.621228]  __rpc_execute+0x85/0x3c0 [sunrpc]
[21650163.621239]  rpc_execute+0xb6/0xd0 [sunrpc]
[21650163.621248]  rpc_run_task+0x144/0x190 [sunrpc]
[21650163.621272]  nfs4_call_sync_custom+0xb/0x30 [nfsv4]
[21650163.621279]  nfs4_do_call_sync+0x69/0x90 [nfsv4]
[21650163.621287]  _nfs4_proc_statfs+0xcb/0xf0 [nfsv4]
[21650163.621296]  nfs4_proc_statfs+0x65/0xa0 [nfsv4]
[21650163.621307]  nfs_statfs+0x6c/0x170 [nfs]
[21650163.621312]  ? __audit_getname+0x29/0x50
[21650163.621315]  statfs_by_dentry+0x67/0x90
[21650163.621316]  vfs_statfs+0x16/0xc0
[21650163.621317]  user_statfs+0x54/0xa0
[21650163.621318]  __do_sys_statfs+0x20/0x50
[21650163.621321]  do_syscall_64+0x5b/0x1a0
[21650163.621324]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[21650163.621326] RIP: 0033:0x48781b
[21650163.621328] Code: e8 2a 35 fe ff eb 88 cc cc cc cc cc cc cc cc e8 db 7a fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[21650163.621329] RSP: 002b:000000c0064b1278 EFLAGS: 00000212 ORIG_RAX: 0000000000000089
[21650163.621329] RAX: ffffffffffffffda RBX: 000000c000062000 RCX: 000000000048781b
[21650163.621330] RDX: 0000000000000000 RSI: 000000c0064b1300 RDI: 000000c002c92a80
[21650163.621331] RBP: 000000c0064b12d8 R08: 00007f0a039c3101 R09: 0000000000000000
[21650163.621331] R10: 00007f099404b310 R11: 0000000000000212 R12: 000000c002c92a80
[21650163.621332] R13: 0000000000000001 R14: 000000c002112d00 R15: ffffffffffffffff
[21650163.621334] Kernel panic - not syncing: softlockup: hung tasks
[21650163.621336] CPU: 51 PID: 39688 Comm: kubelet Kdump: loaded Tainted: G           OEL   --------- -  - 4.18.0-305.el8.x86_64 #1
[21650163.621337] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[21650163.621338] Call Trace:
[21650163.621339]  <IRQ>
[21650163.621344]  dump_stack+0x5c/0x80
[21650163.621348]  panic+0xe7/0x2a9
[21650163.621352]  watchdog_timer_fn.cold.9+0x85/0x9e
[21650163.621354]  ? watchdog+0x30/0x30
[21650163.621356]  __hrtimer_run_queues+0x100/0x280
[21650163.621357]  hrtimer_interrupt+0x100/0x220
[21650163.621360]  smp_apic_timer_interrupt+0x6a/0x130
[21650163.621362]  apic_timer_interrupt+0xf/0x20
[21650163.621363]  </IRQ>
[21650163.621371] RIP: 0010:call_connect+0x0/0x90 [sunrpc]
[21650163.621373] Code: 48 c7 43 20 70 70 9a c0 48 89 df be 3e 00 00 00 5b 5d 41 5c 41 5d 41 5e e9 5d ed 00 00 0f 0b 0f 0b 66 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 53 48 8b 47 30 48 89 fb a8 08 74 52 48 8b 87 d0 00
[21650163.621374] RSP: 0018:ffffa202db95bbb0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[21650163.621375] RAX: 000000000000001d RBX: ffff8a2a5b994800 RCX: 0000000000000001
[21650163.621376] RDX: 000000060a6a603c RSI: ffff8a2a5b994800 RDI: ffff8a2a5b994800
[21650163.621376] RBP: ffff8a28915c91e8 R08: 000000000aaf89aa R09: 000000000aaf89aa
[21650163.621377] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000000
[21650163.621378] R13: 0000000000000000 R14: ffffffffc09b4b20 R15: ffffffffc09a7200
[21650163.621386]  ? call_allocate+0x190/0x190 [sunrpc]
[21650163.621396]  ? __rpc_sleep_on_priority_timeout+0xe0/0xe0 [sunrpc]
[21650163.621406]  __rpc_execute+0x85/0x3c0 [sunrpc]
[21650163.621416]  rpc_execute+0xb6/0xd0 [sunrpc]
[21650163.621424]  rpc_run_task+0x144/0x190 [sunrpc]
[21650163.621431]  nfs4_call_sync_custom+0xb/0x30 [nfsv4]
[21650163.621439]  nfs4_do_call_sync+0x69/0x90 [nfsv4]
[21650163.621446]  _nfs4_proc_statfs+0xcb/0xf0 [nfsv4]
[21650163.621456]  nfs4_proc_statfs+0x65/0xa0 [nfsv4]
[21650163.621463]  nfs_statfs+0x6c/0x170 [nfs]
[21650163.621465]  ? __audit_getname+0x29/0x50
[21650163.621466]  statfs_by_dentry+0x67/0x90
[21650163.621468]  vfs_statfs+0x16/0xc0
[21650163.621469]  user_statfs+0x54/0xa0
[21650163.621470]  __do_sys_statfs+0x20/0x50
[21650163.621472]  do_syscall_64+0x5b/0x1a0
[21650163.621474]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[21650163.621475] RIP: 0033:0x48781b
[21650163.621476] Code: e8 2a 35 fe ff eb 88 cc cc cc cc cc cc cc cc e8 db 7a fe ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[21650163.621477] RSP: 002b:000000c0064b1278 EFLAGS: 00000212 ORIG_RAX: 0000000000000089
[21650163.621478] RAX: ffffffffffffffda RBX: 000000c000062000 RCX: 000000000048781b
[21650163.621479] RDX: 0000000000000000 RSI: 000000c0064b1300 RDI: 000000c002c92a80
[21650163.621480] RBP: 000000c0064b12d8 R08: 00007f0a039c3101 R09: 0000000000000000
[21650163.621480] R10: 00007f099404b310 R11: 0000000000000212 R12: 000000c002c92a80
[21650163.621481] R13: 0000000000000001 R14: 000000c002112d00 R15: ffffffffffffffff