Process seemingly hanging in `irq_work_sync` on kernel-rt with nohz_full

Solution Verified - Updated -

Issue

A process (for example perf) got seemingly stuck/hang with the following backtrace:

crash> bt ffff9e645691c000
PID: 2154866  TASK: ffff9e645691c000  CPU: 35   COMMAND: "perf"
 #0 [fffffe0000702e38] crash_nmi_callback at ffffffff97c52883
 #1 [fffffe0000702e40] nmi_handle at ffffffff97c1f21b
 #2 [fffffe0000702e98] default_do_nmi at ffffffff985ca812
 #3 [fffffe0000702ec0] do_nmi at ffffffff97c1f919
 #4 [fffffe0000702ef0] end_repeat_nmi at ffffffff98601538
    [exception RIP: irq_work_sync+0x4]
    RIP: ffffffff97e28334  RSP: ffffc22d660a7e18  RFLAGS: 00000202
    RAX: 0000000000000023  RBX: ffff9e6df027cea0  RCX: 0000000000000000
    RDX: ffff9e645691c000  RSI: ffff9e5f99fa4250  RDI: ffff9e6df027d1d0
    RBP: ffff9e6df027cea0   R8: 0000000000000000   R9: 0000000000000000
    R10: ffffc22d660a7ea0  R11: 0000000000000000  R12: ffff9e5f99fa4208
    R13: ffffc22d660a7be8  R14: ffff9e6df027cea0  R15: ffff9e6df027d100
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #5 [ffffc22d660a7e18] irq_work_sync at ffffffff97e28334
 #6 [ffffc22d660a7e18] _free_event at ffffffff97e7b8e1
 #7 [ffffc22d660a7e30] perf_event_release_kernel at ffffffff97e7ce0f
 #8 [ffffc22d660a7e90] perf_release at ffffffff97e7ce6c
 #9 [ffffc22d660a7e98] __fput at ffffffff97f46765
#10 [ffffc22d660a7ed8] task_work_run at ffffffff97d10650
#11 [ffffc22d660a7f18] exit_to_usermode_loop at ffffffff97c0264f
#12 [ffffc22d660a7f38] do_syscall_64 at ffffffff97c02e58
#13 [ffffc22d660a7f50] entry_SYSCALL_64_after_hwframe at ffffffff986000a9
    RIP: 00007f3480cd8b47  RSP: 00007ffc4f9a6cc0  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 0000000000000070  RCX: 00007f3480cd8b47
    RDX: 0000000000000000  RSI: 000055d7902992a4  RDI: 0000000000000070
    RBP: 00007ffc4f9a6d00   R8: 000055d790271980   R9: 000055d790271970
    R10: 0000000000000006  R11: 0000000000000293  R12: 0000000000000023
    R13: 000055d78fe90fe0  R14: 0000000000000023  R15: 00007ffc4f9a6d10
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b

Environment

  • Red Hat Enterprise Linux 8
  • kernel-rt

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content