The server is getting highly loaded and slow to respond where node_exporter threads are stuck in busy-wait loop on siglock spinlock
Issue
- The server is getting highly loaded and slow to respond where node_exporter threads are stuck in busy-wait loop on siglock spinlock.
PID: 2628929 TASK: ffffa0efec818000 CPU: 104 COMMAND: "node_exporter"
...
[exception RIP: native_queued_spin_lock_slowpath+324]
RIP: ffffffff9f156154 RSP: ffffad2fcd613df8 RFLAGS: 00000046
RAX: 0000000000000000 RBX: ffffa0f2110f4a08 RCX: 0000000000000051
RDX: ffffa127ffa2bc00 RSI: 0000000001a40000 RDI: ffffa0f2110f4a08
RBP: ffffad2fcd613f58 R8: ffffa0f0afffc538 R9: ffffa0f0f5320480
R10: ffffa0a9c0002780 R11: ffffa0e907aee200 R12: ffffad2fcd613eb0
R13: ffffad2fcd613e90 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#5 [ffffad2fcd613df8] native_queued_spin_lock_slowpath at ffffffff9f156154
#6 [ffffad2fcd613df8] _raw_spin_lock_irq at ffffffff9f9dda85
#7 [ffffad2fcd613e00] get_signal at ffffffff9f10441e
#8 [ffffad2fcd613e58] do_signal at ffffffff9f022ee6
#9 [ffffad2fcd613f20] exit_to_usermode_loop at ffffffff9f003cb9
#10 [ffffad2fcd613f38] do_syscall_64 at ffffffff9f0044ec
#11 [ffffad2fcd613f50] entry_SYSCALL_64_after_hwframe at ffffffff9fa000a9
RIP: 000000000040780e RSP: 000000c0025ab5e8 RFLAGS: 00000206
RAX: fffffffffffffdff RBX: 000000000000002a RCX: 000000000040780e
RDX: 0000000000000080 RSI: 000000c0025ab710 RDI: 000000000000002a
RBP: 000000c0025ab628 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 000000c000463ba0 R15: 00007f5be87a8a5a
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
Environment
- Red Hat Enterprise Linux 8.6.z
- Signal tracing using ftrace
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.