"BUG: soft lockup - CPU#6 stuck for 22s!" messages are reported frequently on worker nodes

Issue

• “BUG: soft lockup - CPU#6 stuck for 22s! [kworker/6:0:3054164]” messages are reported frequently on worker nodes:

Jun 26 09:56:00 worker03 kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/4:2:2146661]
Jun 26 09:56:00 worker03 kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache ceph rbd libceph dns_resolver ip6t_REJECT nf_reject_i
pv6 xt_state xt_owner xt_REDIRECT veth nf_conntrack_netlink nbd xt_recent xt_nat xt_statistic xt_addrtype ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE xt_conntrack xt_comment nft_cou
nter xt_mark nft_compat nft_chain_nat nf_tables overlay vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 
nf_defrag_ipv4 rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ext4 rdma_ucm mbcache ib_umad jbd2 iw_cxgb4 ib_uverbs r
dma_cm iw_cm ib_cm ib_core intel_rapl_msr intel_rapl_common isst_if_common cirrus drm_kms_helper syscopyarea nfit sysfillrect iTCO_wdt sysimgblt iTCO_vendor_support fb_sys_fops l
ibnvdimm joydev virtio_balloon drm i2c_i801 lpc_ich pcspkr ip_tables xfs libcrc32c sr_mod cdrom sg crct10dif_pclmul crc32_pclmul crc32c_intel ahci
Jun 26 09:56:00 worker03 kernel:  libahci virtio_net libata ghash_clmulni_intel virtio_console serio_raw net_failover virtio_blk failover dm_multipath dm_mirror dm
_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse
Jun 26 09:56:00 worker03 kernel: CPU: 4 PID: 2146661 Comm: kworker/4:2 Tainted: G             L   --------- -  - 4.18.0-240.15.1.el8_3.x86_64 #1
Jun 26 09:56:00 worker03 kernel: Hardware name: Baidu Cloud Baidu Cloud BCC, BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
Jun 26 09:56:00 worker03 kernel: Workqueue: events delayed_work [ceph]
Jun 26 09:56:00 worker03 kernel: RIP: 0010:_raw_spin_lock+0x10/0x20
Jun 26 09:56:00 worker03 kernel: Code: 05 48 89 d8 5b c3 e8 5f 99 83 ff eb f4 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 1
7 <85> c0 75 01 c3 89 c6 e8 34 85 83 ff 66 90 c3 90 0f 1f 44 00 00 fa
Jun 26 09:56:00 worker03 kernel: RSP: 0018:ffffb199cd95fe00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Jun 26 09:56:00 worker03 kernel: RAX: 0000000000000000 RBX: ffff92fc894bd7e8 RCX: dead000000000200
Jun 26 09:56:00 worker03 kernel: RDX: 0000000000000001 RSI: ffff92fc894bd640 RDI: ffff92fc894bd870
Jun 26 09:56:00 worker03 kernel: RBP: ffff92fc894bd870 R08: ffff9305063d8df0 R09: 0000000000000002
Jun 26 09:56:00 worker03 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff9305063d8c00
Jun 26 09:56:00 worker03 kernel: R13: ffff9305063d8df0 R14: ffffffffc0eaafb0 R15: ffff92fc894bd7e8
Jun 26 09:56:00 worker03 kernel: FS:  0000000000000000(0000) GS:ffff930b7f500000(0000) knlGS:0000000000000000
Jun 26 09:56:00 worker03 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 26 09:56:00 worker03 kernel: CR2: 00007f89c01ac908 CR3: 0000001052c0a006 CR4: 00000000003606e0
Jun 26 09:56:00 worker03 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 26 09:56:00 worker03 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 26 09:56:00 worker03 kernel: Call Trace:
Jun 26 09:56:00 worker03 kernel:  igrab+0x19/0x50
Jun 26 09:56:00 worker03 kernel:  ceph_check_delayed_caps+0x7b/0x110 [ceph]
Jun 26 09:56:00 worker03 kernel:  delayed_work+0x185/0x290 [ceph]
Jun 26 09:56:00 worker03 kernel:  process_one_work+0x1a7/0x360
Jun 26 09:56:00 worker03 kernel:  worker_thread+0x30/0x390
Jun 26 09:56:00 worker03 kernel:  ? create_worker+0x1a0/0x1a0
Jun 26 09:56:00 worker03 kernel:  kthread+0x112/0x130
Jun 26 09:56:00 worker03 kernel:  ? kthread_flush_work_fn+0x10/0x10
Jun 26 09:56:00 worker03 kernel:  ret_from_fork+0x35/0x40