A soft lockup happens on a CPU on which a task is calling smp_call_function_many() or its variant and waiting for a targeted CPU to handle the IPI

Issue

• A soft lockup happens on a CPU on which a task is calling smp_call_function_many() and waiting for a targeted CPU to handle the IPI.

[26314590.124689] watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [kworker/63:1:1026052]
[26314590.132870] CPU: 63 PID: 1026052 Comm: kworker/63:1 Kdump: loaded Tainted: G        W  OE    --------- -  - 4.18.0-240.el8.x86_64 #1
[26314590.132871] Hardware name: HPE ProLiant DL380 Gen10 Plus/ProLiant DL380 Gen10 Plus, BIOS U46 06/01/2022
[26314590.132877] Workqueue: events netstamp_clear
[26314590.132882] RIP: 0010:smp_call_function_many+0x1ec/0x250
[26314590.132891] Call Trace:
[26314590.132897]  ? poke_int3_handler+0xe0/0xe0
[26314590.132898]  on_each_cpu+0x28/0x60
[26314590.132899]  text_poke_bp_batch+0x8b/0x160
[26314590.132901]  arch_jump_label_transform_apply+0x2e/0x50
[26314590.132905]  static_key_enable_cpuslocked+0x52/0x80
[26314590.132906]  static_key_enable+0x16/0x20
[26314590.132909]  process_one_work+0x1a7/0x360
[26314590.132911]  worker_thread+0x30/0x390
[26314590.132912]  ? create_worker+0x1a0/0x1a0
[26314590.132913]  kthread+0x112/0x130
[26314590.132915]  ? kthread_flush_work_fn+0x10/0x10
[26314590.132918]  ret_from_fork+0x1f/0x40
[26314590.132920] Kernel panic - not syncing: softlockup: hung tasks
[26314590.139058] CPU: 63 PID: 1026052 Comm: kworker/63:1 Kdump: loaded Tainted: G        W  OEL   --------- -  - 4.18.0-240.el8.x86_64 #1
[26314590.151320] Hardware name: HPE ProLiant DL380 Gen10 Plus/ProLiant DL380 Gen10 Plus, BIOS U46 06/01/2022
[26314590.161048] Workqueue: events netstamp_clear
[26314590.165610] Call Trace:
[26314590.168335]  <IRQ>
[26314590.170624]  dump_stack+0x5c/0x80
[26314590.174226]  panic+0xe7/0x2a9
[26314590.177477]  ? __switch_to_asm+0x51/0x70
[26314590.181692]  watchdog_timer_fn.cold.8+0x85/0x9e
[26314590.186518]  ? watchdog+0x30/0x30
[26314590.190120]  __hrtimer_run_queues+0x100/0x280
[26314590.194772]  hrtimer_interrupt+0x100/0x220
[26314590.199160]  smp_apic_timer_interrupt+0x6a/0x130
[26314590.204073]  apic_timer_interrupt+0xf/0x20
[26314590.208461]  </IRQ>
[26314590.210837] RIP: 0010:smp_call_function_many+0x1ec/0x250
[26314590.281718]  ? poke_int3_handler+0xe0/0xe0
[26314590.286562]  ? native_send_call_func_ipi+0xda/0x120
[26314590.292199]  ? poke_int3_handler+0xe0/0xe0
[26314590.297056]  on_each_cpu+0x28/0x60
[26314590.301212]  text_poke_bp_batch+0x8b/0x160
[26314590.306064]  arch_jump_label_transform_apply+0x2e/0x50
[26314590.311964]  static_key_enable_cpuslocked+0x52/0x80
[26314590.317595]  static_key_enable+0x16/0x20
[26314590.322261]  process_one_work+0x1a7/0x360
[26314590.327010]  worker_thread+0x30/0x390
[26314590.331401]  ? create_worker+0x1a0/0x1a0
[26314590.336057]  kthread+0x112/0x130
[26314590.340008]  ? kthread_flush_work_fn+0x10/0x10
[26314590.345177]  ret_from_fork+0x1f/0x40