Deadlock happens between posix_cpu_timer_del() and run_posix_cpu_timers()
Issue
- Deadlock happens between posix_cpu_timer_del() and run_posix_cpu_timers()
- rcu_sched CPU stall and hard lockup messages followed by the crash:
[2280423.826700] INFO: rcu_sched detected stalls on CPUs/tasks: { 16 17 25 27 28 33 42} (detected by 21, t=60010 jiffies, g=550083904, c=550083903, q=846)
[2280423.842272] Task dump for CPU 16:
[2280423.842274] topgun-server.t R running task 0 1794334 2066249 0x00000088
[2280423.842277] Call Trace:
[2280423.845254] [<ffffffffafd8c060>] ? __schedule+0x320/0x680
[2280423.851700] [<ffffffffaf6d7726>] __cond_resched+0x26/0x30
[2280423.858148] [<ffffffffaf867d09>] ? dput+0x29/0x1a0
[2280423.863899] [<ffffffffaf8cd1c4>] ? proc_flush_task+0x174/0x1b0
[2280423.870836] [<ffffffffafd80d82>] ? queued_spin_lock_slowpath+0xb/0xf
[2280423.878374] [<ffffffffaf717d0b>] ? queued_write_lock_slowpath+0x8b/0x90
[2280423.886206] [<ffffffffafd8eedd>] ? _raw_qwrite_lock_irq+0x2d/0x40
[2280423.893444] [<ffffffffaf697745>] ? tasklist_write_lock_irq+0x15/0x20
[2280423.900978] [<ffffffffaf6a0074>] ? release_task+0x44/0x490
[2280423.907517] [<ffffffffaf6a1c7e>] ? do_exit+0x5fe/0xa30
[2280423.913663] [<ffffffffaf71358f>] ? futex_wait+0x11f/0x280
[2280423.920097] [<ffffffffaf6a212f>] ? do_group_exit+0x3f/0xa0
[2280423.926639] [<ffffffffaf6b328e>] ? get_signal_to_deliver+0x1ce/0x5e0
[2280423.934173] [<ffffffffaf62c527>] ? do_signal+0x57/0x6f0
[2280423.940415] [<ffffffffaf7152a6>] ? do_futex+0x106/0x4d0
[2280423.946662] [<ffffffffaf62cc32>] ? do_notify_resume+0x72/0xc0
[2280423.953461] [<ffffffffafd9a2ef>] ? int_signal+0x12/0x17
...
[2280433.674174] NMI watchdog: Watchdog detected hard LOCKUP on cpu 15
[2280433.681123] Modules linked in:
[2280433.692495] scsi_transport_iscsi bonding sunrpc vfat fat dm_queue_length dm_multipath intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel i2c_algo_bit ttm kvm drm_kms_helper syscopyarea sysfillrect sysimgblt irqbypass fb_sys_fops crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ses drm enclosure raid1 glue_helper pcspkr ablk_helper ipmi_si cryptd hpilo joydev sg ipmi_devintf drm_panel_orientation_quirks hpwdt lpc_ich ipmi_msghandler mei_me mei tpm_crb wmi acpi_power_meter onload(OE) binfmt_misc sfc_char(OE) sfc_resource(OE) sfc_affinity(OE) ip_tables xfs libcrc32c dm_snapshot dm_bufio sd_mod lpfc sfc(OE) nvmet_fc nvmet mdio crc_t10dif ptp crct10dif_generic crc32c_intel pps_core crct10dif_pclmul nvme_fc mtd smartpqi nvme_fabrics nvme nvme_core scsi_transport_sas scsi_transport_fc scsi_tgt
[2280433.824683] crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
[2280433.840271] CPU: 15 PID: 1794105 Comm: topgun-server.t Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
[2280433.875243] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/28/2021
[2280433.895785] task: ffff933680a4d280 ti: ffff93410d0d0000 task.ti: ffff93410d0d0000
[2280433.915139] RIP: 0010:[<ffffffffaf717aa2>] [<ffffffffaf717aa2>] native_queued_spin_lock_slowpath+0x122/0x200
[2280433.937759] RSP: 0018:ffff93410d0d3c20 EFLAGS: 00000046
[2280433.955057] RAX: 0000000000000000 RBX: ffffffffb0207080 RCX: 0000000000790000
[2280433.974419] RDX: ffff936e3fadb8c0 RSI: 0000000000190001 RDI: ffffffffb0207084
[2280433.994089] RBP: ffff93410d0d3c20 R08: ffff93af3f45b8c0 R09: 0000000000000000
[2280434.013812] R10: 0000000000000001 R11: ffff93ae167e7f00 R12: ffffffffb0207084
[2280434.033302] R13: ffff933680a4d280 R14: ffff933680a4d280 R15: ffff93410d0d3858
[2280434.052801] FS: 00007f11f75fe700(0000) GS:ffff93af3f440000(0000) knlGS:0000000000000000
[2280434.073810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2280434.093236] CR2: 00007fb220093ff8 CR3: 0000006205c10000 CR4: 00000000007607e0
[2280434.113541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2280434.133873] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2280434.153909] PKRU: 55555554
[2280434.168814] Call Trace:
[2280434.183316] [<ffffffffafd80d82>] queued_spin_lock_slowpath+0xb/0xf
[2280434.202247] [<ffffffffaf717d0b>] queued_write_lock_slowpath+0x8b/0x90
[2280434.221511] [<ffffffffafd8eedd>] _raw_qwrite_lock_irq+0x2d/0x40
[2280434.240558] [<ffffffffaf697745>] tasklist_write_lock_irq+0x15/0x20
[2280434.259854] [<ffffffffaf6a0074>] release_task+0x44/0x490
[2280434.277920] [<ffffffffaf6a1c7e>] do_exit+0x5fe/0xa30
[2280434.295414] [<ffffffffaf71358f>] ? futex_wait+0x11f/0x280
[2280434.313412] [<ffffffffaf6594b2>] ? native_smp_send_reschedule+0x52/0x70
[2280434.333275] [<ffffffffaf6a212f>] do_group_exit+0x3f/0xa0
[2280434.351298] [<ffffffffaf6b328e>] get_signal_to_deliver+0x1ce/0x5e0
[2280434.370347] [<ffffffffaf62c527>] do_signal+0x57/0x6f0
[2280434.388293] [<ffffffffaf7152a6>] ? do_futex+0x106/0x4d0
[2280434.406302] [<ffffffffaf6b21b5>] ? do_send_specific+0x75/0xa0
[2280434.424902] [<ffffffffaf6b227f>] ? do_tkill+0x9f/0xd0
[2280434.443045] [<ffffffffaf62cc32>] do_notify_resume+0x72/0xc0
[2280434.463626] [<ffffffffafd9a2ef>] int_signal+0x12/0x17
[2280434.481800] Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 60 18 35 b0 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
[2280434.528135] Kernel panic - not syncing: Hard LOCKUP
Environment
- Red Hat Enterprise Linux 7.9.z - kernel-3.10.0-1160.59.1.el7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.