A significant number of soft lockup occurrences, leading to a server hang-up

Issue

• A significant number of soft lockup occurrences, leading to a server hang-up

Feb 24 14:14:11 hostname kernel: watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [CPU 2/KVM:455201]
Feb 24 14:14:11 hostname kernel: Modules linked in: bmhook(OE) veth nf_conntrack_netlink vhost_net vhost vhost_iotlb tap tun tcp_diag udp_diag inet_diag joydev binfmt_misc tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) sha512_ssse3 sha512_generic overlay sch_ingress bonding geneve ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout openvswitch nf_conncount nf_nat nls_utf8 isofs cdrom nf_log_ipv4 xt_CT nf_log_ipv6 nf_log_common nft_limit nft_counter xt_LOG xt_limit xt_multiport xt_comment xt_conntrack nft_compat nf_tables nfnetlink rpcrdma sunrpc rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi ib_umad scsi_transport_iscsi ib_ipoib rdma_cm iw_cm ib_cm intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass mlx5_ib crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl ib_uverbs intel_cstate ib_core acpi_ipmi ipmi_si intel_uncore ses ioatdma pcspkr ipmi_devintf hpwdt enclosure wmi lpc_ich mei_me
Feb 24 14:14:11 hostname kernel:  acpi_power_meter mei hpilo ipmi_msghandler dca acpi_tad xfs sd_mod t10_pi sg mlx5_core mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm serio_raw smartpqi tg3 mlxfw scsi_transport_sas pci_hyperv_intf tls i2c_algo_bit uas usb_storage fuse nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_intel br_netfilter bridge stp llc [last unloaded: bmhook]
Feb 24 14:14:11 hostname kernel: Features: eBPF/event
Feb 24 14:14:11 hostname kernel: CPU: 25 PID: 455201 Comm: CPU 2/KVM Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-305.88.1.el8_4.x86_64 #1
Feb 24 14:14:11 hostname kernel: Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 03/16/2023
Feb 24 14:14:11 hostname kernel: RIP: 0010:smp_call_function_single+0xd2/0xf0
Feb 24 14:14:11 hostname kernel: Code: 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 7d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 8b 54 24 18 <83> e2 01 75 f5 eb ca 8b 05 81 3c d3 01 85 c0 75 88 0f 0b eb 84 e8 
Feb 24 14:14:11 hostname kernel: RSP: 0018:ffff9ffee3bebbe0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Feb 24 14:14:11 hostname kernel: RAX: 0000000000000000 RBX: ffff9479f7238000 RCX: 0000000000000830
Feb 24 14:14:11 hostname kernel: RDX: 0000000000000003 RSI: 00000000000008fb RDI: 0000000000000830
Feb 24 14:14:11 hostname kernel: RBP: ffff9ffee3bebc38 R08: 00000000001199b8 R09: 0000000000118c88
Feb 24 14:14:11 hostname kernel: R10: 0000000000000000 R11: 000000000000027d R12: 0000000000000019
Feb 24 14:14:11 hostname kernel: R13: 0000000000000000 R14: 000000000000002c R15: 0000000000000019
Feb 24 14:14:11 hostname kernel: FS:  00007f277f5fe700(0000) GS:ffff947f3f040000(0000) knlGS:0000000000000000
Feb 24 14:14:11 hostname kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
Feb 24 14:14:11 hostname kernel: CR2: 000055ce0a6c3438 CR3: 0000007a5faa2002 CR4: 00000000007726e0
Feb 24 14:14:11 hostname kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 24 14:14:11 hostname kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Feb 24 14:14:11 hostname kernel: PKRU: 55555554
Feb 24 14:14:11 hostname kernel: Call Trace:
Feb 24 14:14:11 hostname kernel:  ? crash_vmclear_local_loaded_vmcss+0xe0/0xe0 [kvm_intel]
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x41/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x41/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  vmx_vcpu_load_vmcs+0x1a2/0x450 [kvm_intel]
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x41/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x41/0x70
Feb 24 14:14:11 hostname kernel:  ? __switch_to_asm+0x35/0x70
Feb 24 14:14:11 hostname kernel:  vmx_vcpu_load+0x13/0x40 [kvm_intel]
Feb 24 14:14:11 hostname kernel:  kvm_arch_vcpu_load+0x5a/0x230 [kvm]
Feb 24 14:14:11 hostname kernel:  finish_task_switch+0x1f8/0x2f0
Feb 24 14:14:11 hostname kernel:  __schedule+0x2d5/0x7b0
Feb 24 14:14:11 hostname kernel:  ? kvm_apic_has_interrupt+0x44/0x80 [kvm]
Feb 24 14:14:11 hostname kernel:  schedule+0x38/0xa0
Feb 24 14:14:11 hostname kernel:  kvm_vcpu_block+0x59/0x320 [kvm]
Feb 24 14:14:11 hostname kernel:  kvm_arch_vcpu_ioctl_run+0x194/0x520 [kvm]
Feb 24 14:14:11 hostname kernel:  kvm_vcpu_ioctl+0x22a/0x5e0 [kvm]
Feb 24 14:14:11 hostname kernel:  do_vfs_ioctl+0xa4/0x680
Feb 24 14:14:11 hostname kernel:  ksys_ioctl+0x60/0x90
Feb 24 14:14:11 hostname kernel:  __x64_sys_ioctl+0x16/0x20
Feb 24 14:14:11 hostname kernel:  do_syscall_64+0x5b/0x1a0
Feb 24 14:14:11 hostname kernel:  entry_SYSCALL_64_after_hwframe+0x65/0xca
Feb 24 14:14:11 hostname kernel: RIP: 0033:0x7f38ae4ee62b
Feb 24 14:14:11 hostname kernel: Code: 0f 1e fa 48 8b 05 5d b8 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 2d b8 2c 00 f7 d8 64 89 01 48