The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()

Solution Verified - Updated -

Issue

  • The server crashes due to a soft lockup that occurs on one CPU on which a task is looping in a tight loop in csd_lock_wait()
[48166.961710] Kernel panic - not syncing: softlockup: hung tasks
[48166.961727] CPU: 69 PID: 16170 Comm: node_exporter Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
[48166.961758] Hardware name: Dell Inc. PowerEdge R640/0RGP26, BIOS 2.12.2 07/09/2021
[48166.961777] Call Trace:
[48166.961784]  <IRQ>  [<ffffffffaf3865c9>] dump_stack+0x19/0x1b
[48166.961803]  [<ffffffffaf3802d1>] panic+0xe8/0x21f
[48166.961819]  [<ffffffffaed4ef0a>] watchdog_timer_fn+0x20a/0x220
[48166.961834]  [<ffffffffaed4ed00>] ? watchdog+0x40/0x40
[48166.961849]  [<ffffffffaecca38e>] __hrtimer_run_queues+0x10e/0x270
[48166.961865]  [<ffffffffaecca8ef>] hrtimer_interrupt+0xaf/0x1d0
[48166.961882]  [<ffffffffaec5cf0b>] local_apic_timer_interrupt+0x3b/0x60
[48166.961901]  [<ffffffffaf39ea23>] smp_apic_timer_interrupt+0x43/0x60
[48166.961918]  [<ffffffffaf39afba>] apic_timer_interrupt+0x16a/0x170
[48166.961933]  <EOI>  [<ffffffffaeddea1d>] ? kvmalloc_node+0x8d/0xe0
[48166.961954]  [<ffffffffaed16ae6>] ? generic_exec_single+0x106/0x1c0
[48166.961971]  [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.961988]  [<ffffffffaefbbeb0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[48166.962005]  [<ffffffffaed16bff>] smp_call_function_single+0x5f/0xa0
[48166.962022]  [<ffffffffaefbc00d>] rdmsr_on_cpu+0x5d/0x90
[48166.962037]  [<ffffffffc0469327>] show_crit_alarm+0x47/0x80 [coretemp]
[48166.962056]  [<ffffffffaee28d8c>] ? __kmalloc_node+0x5c/0x2b0
[48166.962073]  [<ffffffffaf0b6ee3>] dev_attr_show+0x23/0x60
[48166.962088]  [<ffffffffaf38a832>] ? mutex_lock+0x12/0x2f
[48166.962104]  [<ffffffffaeedbd4f>] sysfs_kf_seq_show+0xcf/0x1f0
[48166.962120]  [<ffffffffaeeda406>] kernfs_seq_show+0x26/0x30
[48166.962135]  [<ffffffffaee76dc0>] seq_read+0x130/0x450
[48166.962149]  [<ffffffffaeedad65>] kernfs_fop_read+0x105/0x170
[48166.962166]  [<ffffffffaee4e31f>] vfs_read+0x9f/0x170
[48166.962180]  [<ffffffffaee4f165>] SyS_read+0x55/0xd0
[48166.962742]  [<ffffffffaf399f92>] system_call_fastpath+0x25/0x2a

Environment

  • Red Hat Enterprise Linux 7.9.z
    • kernel-3.10.0-1160.76.1.el7
  • (Hardware) Dell PowerEdge R640
    • Please note, this sort of issues can happen with any server models provided from any hardware manufacturers.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content