A soft lockup happens on a CPU on which a task is calling smp_call_function_many() and waiting for a targeted CPU to handle the IPI

Issue

• A soft lockup happens on a CPU on which a task is calling smp_call_function_many() and waiting for a targeted CPU to handle the IPI.

[26314590.124689] watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [kworker/63:1:1026052]
[26314590.132847] Modules linked in: nf_tables nfnetlink sctp sit tunnel4 ip_tunnel 8021q garp mrp stp llc bonding intel_rapl_msr intel_rapl_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_uncore intel_rapl_perf ses pcspkr ipmi_si enclosure isst_if_mbox_pci ipmi_devintf isst_if_mmio scsi_transport_sas isst_if_common hpilo hpwdt ioatdma ipmi_msghandler acpi_tad wmi acpi_power_meter ip_tables xfs libcrc32c sd_mod sg mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_vram_helper drm_ttm_helper ttm drm igb crc32c_intel serio_raw ice(OE) megaraid_sas dca auxiliary(OE) i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod
[26314590.132870] CPU: 63 PID: 1026052 Comm: kworker/63:1 Kdump: loaded Tainted: G        W  OE    --------- -  - 4.18.0-240.el8.x86_64 #1
[26314590.132871] Hardware name: HPE ProLiant DL380 Gen10 Plus/ProLiant DL380 Gen10 Plus, BIOS U46 06/01/2022
[26314590.132877] Workqueue: events netstamp_clear
[26314590.132882] RIP: 0010:smp_call_function_many+0x1ec/0x250
[26314590.132884] Code: c7 e8 08 8a 75 00 3b 05 d6 f9 2b 01 0f 83 99 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 60 e8 15 88 8b 51 18 83 e2 01 74 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c7 48 c7 c2 40 57 41 88 4c 89 f6 89 df
[26314590.132884] RSP: 0018:ff8297214db8fdd0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[26314590.132885] RAX: 0000000000000003 RBX: ff3ab41abfdeae80 RCX: ff3ab41abf4f21a0
[26314590.132886] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ff3ab3dc4fcb96f0
[26314590.132886] RBP: ffffffff87027610 R08: 000000000002f060 R09: ffffffff8704fc8a
[26314590.132887] R10: ffe1d2c6933e2980 R11: 0000000000000001 R12: 0000000000000000
[26314590.132887] R13: 0000000000000001 R14: 0000000000000060 R15: 0000000000000001
[26314590.132888] FS:  0000000000000000(0000) GS:ff3ab41abfdc0000(0000) knlGS:0000000000000000
[26314590.132888] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26314590.132889] CR2: 00007fffa1fac888 CR3: 0000004487a0a005 CR4: 0000000000761ee0
[26314590.132889] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26314590.132889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[26314590.132890] PKRU: 55555554
[26314590.132891] Call Trace:
[26314590.132897]  ? poke_int3_handler+0xe0/0xe0
[26314590.132898]  on_each_cpu+0x28/0x60
[26314590.132899]  text_poke_bp_batch+0x8b/0x160
[26314590.132901]  arch_jump_label_transform_apply+0x2e/0x50
[26314590.132905]  static_key_enable_cpuslocked+0x52/0x80
[26314590.132906]  static_key_enable+0x16/0x20
[26314590.132909]  process_one_work+0x1a7/0x360
[26314590.132911]  worker_thread+0x30/0x390
[26314590.132912]  ? create_worker+0x1a0/0x1a0
[26314590.132913]  kthread+0x112/0x130
[26314590.132915]  ? kthread_flush_work_fn+0x10/0x10
[26314590.132918]  ret_from_fork+0x1f/0x40
[26314590.132920] Kernel panic - not syncing: softlockup: hung tasks
[26314590.139058] CPU: 63 PID: 1026052 Comm: kworker/63:1 Kdump: loaded Tainted: G        W  OEL   --------- -  - 4.18.0-240.el8.x86_64 #1
[26314590.151320] Hardware name: HPE ProLiant DL380 Gen10 Plus/ProLiant DL380 Gen10 Plus, BIOS U46 06/01/2022
[26314590.161048] Workqueue: events netstamp_clear
[26314590.165610] Call Trace:
[26314590.168335]  <IRQ>
[26314590.170624]  dump_stack+0x5c/0x80
[26314590.174226]  panic+0xe7/0x2a9
[26314590.177477]  ? __switch_to_asm+0x51/0x70
[26314590.181692]  watchdog_timer_fn.cold.8+0x85/0x9e
[26314590.186518]  ? watchdog+0x30/0x30
[26314590.190120]  __hrtimer_run_queues+0x100/0x280
[26314590.194772]  hrtimer_interrupt+0x100/0x220
[26314590.199160]  smp_apic_timer_interrupt+0x6a/0x130
[26314590.204073]  apic_timer_interrupt+0xf/0x20
[26314590.208461]  </IRQ>
[26314590.210837] RIP: 0010:smp_call_function_many+0x1ec/0x250
[26314590.216450] Code: c7 e8 08 8a 75 00 3b 05 d6 f9 2b 01 0f 83 99 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 60 e8 15 88 8b 51 18 83 e2 01 74 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c7 48 c7 c2 40 57 41 88 4c 89 f6 89 df
[26314590.235629] RSP: 0018:ff8297214db8fdd0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[26314590.243516] RAX: 0000000000000003 RBX: ff3ab41abfdeae80 RCX: ff3ab41abf4f21a0
[26314590.250967] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ff3ab3dc4fcb96f0
[26314590.258416] RBP: ffffffff87027610 R08: 000000000002f060 R09: ffffffff8704fc8a
[26314590.265865] R10: ffe1d2c6933e2980 R11: 0000000000000001 R12: 0000000000000000
[26314590.273803] R13: 0000000000000001 R14: 0000000000000060 R15: 0000000000000001
[26314590.281718]  ? poke_int3_handler+0xe0/0xe0
[26314590.286562]  ? native_send_call_func_ipi+0xda/0x120
[26314590.292199]  ? poke_int3_handler+0xe0/0xe0
[26314590.297056]  on_each_cpu+0x28/0x60
[26314590.301212]  text_poke_bp_batch+0x8b/0x160
[26314590.306064]  arch_jump_label_transform_apply+0x2e/0x50
[26314590.311964]  static_key_enable_cpuslocked+0x52/0x80
[26314590.317595]  static_key_enable+0x16/0x20
[26314590.322261]  process_one_work+0x1a7/0x360
[26314590.327010]  worker_thread+0x30/0x390
[26314590.331401]  ? create_worker+0x1a0/0x1a0
[26314590.336057]  kthread+0x112/0x130
[26314590.340008]  ? kthread_flush_work_fn+0x10/0x10
[26314590.345177]  ret_from_fork+0x1f/0x40