The server is getting highly loaded and slow to respond where node_exporter threads are stuck in busy-wait loop on siglock spinlock

Solution Verified - Updated -

Issue

  • The server is getting highly loaded and slow to respond where node_exporter threads are stuck in busy-wait loop on siglock spinlock.
PID: 2628929  TASK: ffffa0efec818000  CPU: 104  COMMAND: "node_exporter"
        ...
    [exception RIP: native_queued_spin_lock_slowpath+324]
    RIP: ffffffff9f156154  RSP: ffffad2fcd613df8  RFLAGS: 00000046
    RAX: 0000000000000000  RBX: ffffa0f2110f4a08  RCX: 0000000000000051
    RDX: ffffa127ffa2bc00  RSI: 0000000001a40000  RDI: ffffa0f2110f4a08
    RBP: ffffad2fcd613f58   R8: ffffa0f0afffc538   R9: ffffa0f0f5320480
    R10: ffffa0a9c0002780  R11: ffffa0e907aee200  R12: ffffad2fcd613eb0
    R13: ffffad2fcd613e90  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #5 [ffffad2fcd613df8] native_queued_spin_lock_slowpath at ffffffff9f156154
 #6 [ffffad2fcd613df8] _raw_spin_lock_irq at ffffffff9f9dda85
 #7 [ffffad2fcd613e00] get_signal at ffffffff9f10441e
 #8 [ffffad2fcd613e58] do_signal at ffffffff9f022ee6
 #9 [ffffad2fcd613f20] exit_to_usermode_loop at ffffffff9f003cb9
#10 [ffffad2fcd613f38] do_syscall_64 at ffffffff9f0044ec
#11 [ffffad2fcd613f50] entry_SYSCALL_64_after_hwframe at ffffffff9fa000a9
    RIP: 000000000040780e  RSP: 000000c0025ab5e8  RFLAGS: 00000206
    RAX: fffffffffffffdff  RBX: 000000000000002a  RCX: 000000000040780e
    RDX: 0000000000000080  RSI: 000000c0025ab710  RDI: 000000000000002a
    RBP: 000000c0025ab628   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000206  R12: 0000000000000000
    R13: 0000000000000000  R14: 000000c000463ba0  R15: 00007f5be87a8a5a
    ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b

Environment

  • Red Hat Enterprise Linux 8.6.z
  • Signal tracing using ftrace

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content