The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()
Issue
- The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()
[48166.961710] Kernel panic - not syncing: softlockup: hung tasks
[48166.961727] CPU: 69 PID: 16170 Comm: node_exporter Kdump: loaded Tainted: G L ------------ 3.10.0-1160.76.1.el7.x86_64 #1
[48166.961758] Hardware name: Dell Inc. PowerEdge R640/0RGP26, BIOS 2.12.2 07/09/2021
[48166.961777] Call Trace:
[48166.961784] <IRQ> [<ffffffffaf3865c9>] dump_stack+0x19/0x1b
[48166.961803] [<ffffffffaf3802d1>] panic+0xe8/0x21f
[48166.961819] [<ffffffffaed4ef0a>] watchdog_timer_fn+0x20a/0x220
[48166.961834] [<ffffffffaed4ed00>] ? watchdog+0x40/0x40
[48166.961849] [<ffffffffaecca38e>] __hrtimer_run_queues+0x10e/0x270
[48166.961865] [<ffffffffaecca8ef>] hrtimer_interrupt+0xaf/0x1d0
[48166.961882] [<ffffffffaec5cf0b>] local_apic_timer_interrupt+0x3b/0x60
[48166.961901] [<ffffffffaf39ea23>] smp_apic_timer_interrupt+0x43/0x60
[48166.961918] [<ffffffffaf39afba>] apic_timer_interrupt+0x16a/0x170
[48166.961933] <EOI> [<ffffffffaeddea1d>] ? kvmalloc_node+0x8d/0xe0
[48166.961954] [<ffffffffaed16ae6>] ? generic_exec_single+0x106/0x1c0
[48166.961971] [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.961988] [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.962005] [<ffffffffaed16bff>] smp_call_function_single+0x5f/0xa0
[48166.962022] [<ffffffffaefbc00d>] rdmsr_on_cpu+0x5d/0x90
[48166.962037] [<ffffffffc0469327>] show_crit_alarm+0x47/0x80 [coretemp]
[48166.962056] [<ffffffffaee28d8c>] ? __kmalloc_node+0x5c/0x2b0
[48166.962073] [<ffffffffaf0b6ee3>] dev_attr_show+0x23/0x60
[48166.962088] [<ffffffffaf38a832>] ? mutex_lock+0x12/0x2f
[48166.962104] [<ffffffffaeedbd4f>] sysfs_kf_seq_show+0xcf/0x1f0
[48166.962120] [<ffffffffaeeda406>] kernfs_seq_show+0x26/0x30
[48166.962135] [<ffffffffaee76dc0>] seq_read+0x130/0x450
[48166.962149] [<ffffffffaeedad65>] kernfs_fop_read+0x105/0x170
[48166.962166] [<ffffffffaee4e31f>] vfs_read+0x9f/0x170
[48166.962180] [<ffffffffaee4f165>] SyS_read+0x55/0xd0
[48166.962742] [<ffffffffaf399f92>] system_call_fastpath+0x25/0x2a
Environment
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 8
- (Hardware) Dell PowerEdge Servers
- Please be aware that such issues can occur, not just limited to specific server models.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.