Skip to navigation

Systems hang due to leap-second livelock.

Updated 2013-08-12T20:16:55+00:00

Issue

  • Systems repeatedly crash due to NMI Watchdog detecting a hang. There are two similar backtraces that appear to be related to this issue.
  1. The first backtrace appears to be in ktime_get:
PID: 15173  TASK: ffff88065c968ac0  CPU: 5   COMMAND: "java"
 #0 [ffff880685447b00] machine_kexec at ffffffff810321cb
 #1 [ffff880685447b60] crash_kexec at ffffffff810b8ef2
 #2 [ffff880685447c30] panic at ffffffff814ec3d1
 #3 [ffff880685447cb0] watchdog_overflow_callback at ffffffff810d8f2d
 #4 [ffff880685447cd0] __perf_event_overflow at ffffffff8110a81d
 #5 [ffff880685447d70] perf_event_overflow at ffffffff8110add4
 #6 [ffff880685447d80] intel_pmu_handle_irq at ffffffff8101e296
 #7 [ffff880685447e90] perf_event_nmi_handler at ffffffff814f0a89
 #8 [ffff880685447ea0] notifier_call_chain at ffffffff814f25d5
 #9 [ffff880685447ee0] atomic_notifier_call_chain at ffffffff814f263a
#10 [ffff880685447ef0] notify_die at ffffffff81096a3e
#11 [ffff880685447f20] do_nmi at ffffffff814f0253
#12 [ffff880685447f50] nmi at ffffffff814efb60
    [exception RIP: ktime_get+180]
    RIP: ffffffff8109b764  RSP: ffff880685443ee8  RFLAGS: 00000002
    RAX: 0000000000010f00  RBX: ffff880685450f08  RCX: 0000000000000000
    RDX: ffff880685440000  RSI: 0000000000000005  RDI: ffff88068544db00
    RBP: ffff880685443f08   R8: 7fffffffffffffff   R9: 0000000000000001
    R10: 0000000000000018  R11: 00007f6790cc6620  R12: ffff880685443f48
    R13: 000000005fe553fd  R14: 0000000000000000  R15: 7fffffffffffffff
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
#13 [ffff880685443ee8] ktime_get at ffffffff8109b764
#14 [ffff880685443f10] hrtimer_interrupt at ffffffff81095509
#15 [ffff880685443f90] smp_apic_timer_interrupt at ffffffff814f4e8b
#16 [ffff880685443fb0] apic_timer_interrupt at ffffffff8100bc13
--- <IRQ stack> ---
#17 [ffff88065c3d3e28] apic_timer_interrupt at ffffffff8100bc13
    [exception RIP: current_kernel_time+59]
    RIP: ffffffff8109b7cb  RSP: ffff88065c3d3ed8  RFLAGS: 00000202
    RAX: 0000000000000001  RBX: ffff88065c3d3ef8  RCX: 000000005fe553fd
    RDX: 00007f6e5d3f4500  RSI: 0000000000000000  RDI: 00000000c000003e
    RBP: ffffffff8100bc0e   R8: 0000000000000000   R9: 00007f6e548f7890
    R10: 0000000000000018  R11: 00007f6790cc6620  R12: 0000000000000000
    R13: ffff88065c968af8  R14: ffff88065c968ac0  R15: 0000000000000000
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#18 [ffff88065c3d3f00] audit_syscall_entry at ffffffff810d4662
#19 [ffff88065c3d3f80] auditsys at ffffffff8100b1dd
    RIP: 00000031df0cdc87  RSP: 00007f6e548f7958  RFLAGS: 00000207
    RAX: 0000000000000018  RBX: ffffffff8100b1dd  RCX: 0000000000000f8c
    RDX: 0000000000000000  RSI: 00007f6e5d3f4500  RDI: 00007f6790cc6620
    RBP: 00007f6e548f7940   R8: 0000000000002000   R9: 0000000000003b45
    R10: 00007f6e548f7890  R11: 0000000000000202  R12: 0000000000000000
    R13: 00000000000001d1  R14: 0000000000000004  R15: 00007f6790cc6620
    ORIG_RAX: 0000000000000018  CS: 0033  SS: 002b
  1. The second backtrace appears to be in getnstimeofday:
PID: 0      TASK: ffff880660799540  CPU: 11  COMMAND: "swapper"
 #0 [ffff8806854a7b00] machine_kexec at ffffffff810321cb
 #1 [ffff8806854a7b60] crash_kexec at ffffffff810b8ef2
 #2 [ffff8806854a7c30] panic at ffffffff814ec3d1
 #3 [ffff8806854a7cb0] watchdog_overflow_callback at ffffffff810d8f2d
 #4 [ffff8806854a7cd0] __perf_event_overflow at ffffffff8110a81d
 #5 [ffff8806854a7d70] perf_event_overflow at ffffffff8110add4
 #6 [ffff8806854a7d80] intel_pmu_handle_irq at ffffffff8101e296
 #7 [ffff8806854a7e90] perf_event_nmi_handler at ffffffff814f0a89
 #8 [ffff8806854a7ea0] notifier_call_chain at ffffffff814f25d5
 #9 [ffff8806854a7ee0] atomic_notifier_call_chain at ffffffff814f263a
#10 [ffff8806854a7ef0] notify_die at ffffffff81096a3e
#11 [ffff8806854a7f20] do_nmi at ffffffff814f0253
#12 [ffff8806854a7f50] nmi at ffffffff814efb60
    [exception RIP: getnstimeofday+202]
    RIP: ffffffff8109b34a  RSP: ffff880c60455e18  RFLAGS: 00000002
    RAX: 0000000000000000  RBX: ffff880c60455e48  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: ffff880c60455fd8  RDI: ffff880c60455e48
    RBP: ffff880c60455e38   R8: 0000000000000003   R9: 0000000000000320
    R10: 001ebf406e42bee9  R11: 0000000000000000  R12: 00000000a0a2068b
    R13: 129c6a9e932bdae1  R14: 0000000000000002  R15: 000000000000000b
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
#13 [ffff880c60455e18] getnstimeofday at ffffffff8109b34a
#14 [ffff880c60455e40] ktime_get_real at ffffffff8109b386
#15 [ffff880c60455e60] intel_idle at ffffffff812c4ab6
#16 [ffff880c60455ee0] cpuidle_idle_call at ffffffff813f9ff7
#17 [ffff880c60455f00] cpu_idle at ffffffff81009e06

Environment

  • Red Hat Enterprise Linux 6

Subscriber content preview. For full access to the Red Hat Knowledgebase, please log in.

Not a subscriber? Learn more about the benefits of Red Hat Subscriptions.