The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()

Solution Verified - Updated -

Issue

  • The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()
[48166.961710] Kernel panic - not syncing: softlockup: hung tasks
[48166.961727] CPU: 69 PID: 16170 Comm: node_exporter Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
[48166.961758] Hardware name: Dell Inc. PowerEdge R640/0RGP26, BIOS 2.12.2 07/09/2021
[48166.961777] Call Trace:
[48166.961784]  <IRQ>  [<ffffffffaf3865c9>] dump_stack+0x19/0x1b
[48166.961803]  [<ffffffffaf3802d1>] panic+0xe8/0x21f
[48166.961819]  [<ffffffffaed4ef0a>] watchdog_timer_fn+0x20a/0x220
[48166.961834]  [<ffffffffaed4ed00>] ? watchdog+0x40/0x40
[48166.961849]  [<ffffffffaecca38e>] __hrtimer_run_queues+0x10e/0x270
[48166.961865]  [<ffffffffaecca8ef>] hrtimer_interrupt+0xaf/0x1d0
[48166.961882]  [<ffffffffaec5cf0b>] local_apic_timer_interrupt+0x3b/0x60
[48166.961901]  [<ffffffffaf39ea23>] smp_apic_timer_interrupt+0x43/0x60
[48166.961918]  [<ffffffffaf39afba>] apic_timer_interrupt+0x16a/0x170
[48166.961933]  <EOI>  [<ffffffffaeddea1d>] ? kvmalloc_node+0x8d/0xe0
[48166.961954]  [<ffffffffaed16ae6>] ? generic_exec_single+0x106/0x1c0
[48166.961971]  [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.961988]  [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.962005]  [<ffffffffaed16bff>] smp_call_function_single+0x5f/0xa0
[48166.962022]  [<ffffffffaefbc00d>] rdmsr_on_cpu+0x5d/0x90
[48166.962037]  [<ffffffffc0469327>] show_crit_alarm+0x47/0x80 [coretemp]
[48166.962056]  [<ffffffffaee28d8c>] ? __kmalloc_node+0x5c/0x2b0
[48166.962073]  [<ffffffffaf0b6ee3>] dev_attr_show+0x23/0x60
[48166.962088]  [<ffffffffaf38a832>] ? mutex_lock+0x12/0x2f
[48166.962104]  [<ffffffffaeedbd4f>] sysfs_kf_seq_show+0xcf/0x1f0
[48166.962120]  [<ffffffffaeeda406>] kernfs_seq_show+0x26/0x30
[48166.962135]  [<ffffffffaee76dc0>] seq_read+0x130/0x450
[48166.962149]  [<ffffffffaeedad65>] kernfs_fop_read+0x105/0x170
[48166.962166]  [<ffffffffaee4e31f>] vfs_read+0x9f/0x170
[48166.962180]  [<ffffffffaee4f165>] SyS_read+0x55/0xd0
[48166.962742]  [<ffffffffaf399f92>] system_call_fastpath+0x25/0x2a

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • (Hardware) Dell PowerEdge Servers
    • Please be aware that such issues can occur, not just limited to specific server models.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content