Kernel Panic with Hard Lockup or Soft Lockups in getnstimeofday64

Solution Verified - Updated -

Issue

  • The system panics with a hard lockup while a CPU is at __getnstimeofday64+0x90
  • A hard lockup is experienced where the offending CPU may have been at getnstimeofday64+0xe in an interrupt

    [1459541.125835] Kernel panic - not syncing: Hard LOCKUP
    [1459541.126109] CPU: 11 PID: 2632 Comm: X Tainted: P           OEL ------------   3.10.0-693.11.6.el7.x86_64 #1
    [1459541.126606] Hardware name: Cisco Systems Inc UCSC-C220-M4S/UCSC-C220-M4S, BIOS C220M4.3.0.4a.0.0226182253 02/26/2018
    [1459541.127136] Call Trace:
    [1459541.127272]  <NMI>  [<ffffffff816a5ea1>] dump_stack+0x19/0x1b
    [1459541.127609]  [<ffffffff8169fd64>] panic+0xe8/0x20d
    [1459541.127874]  [<ffffffff8108919f>] nmi_panic+0x3f/0x40
    [1459541.128146]  [<ffffffff81131775>] watchdog_overflow_callback+0xf5/0x100
    [1459541.128495]  [<ffffffff81170261>] __perf_event_overflow+0x51/0xf0
    [1459541.128820]  [<ffffffff81178db4>] perf_event_overflow+0x14/0x20
    [1459541.129136]  [<ffffffff81009f88>] intel_pmu_handle_irq+0x218/0x4f0
    [1459541.129467]  [<ffffffff8132e4c2>] ? put_dec+0x72/0x90
    [1459541.129739]  [<ffffffff8132f4b3>] ? number.isra.2+0x323/0x360
    [1459541.130046]  [<ffffffff81326c6c>] ? ioremap_page_range+0x26c/0x3d0
    [1459541.130372]  [<ffffffff811c28c4>] ? vunmap_page_range+0x1b4/0x300
    [1459541.130700]  [<ffffffff813dc306>] ? ghes_copy_tofrom_phys+0x116/0x210
    [1459541.131041]  [<ffffffff813dc4a0>] ? ghes_read_estatus+0xa0/0x190
    [1459541.131363]  [<ffffffff816af02b>] perf_event_nmi_handler+0x2b/0x50
    [1459541.131692]  [<ffffffff816b0c57>] nmi_handle.isra.0+0x87/0x160
    [1459541.132004]  [<ffffffff816b0f40>] do_nmi+0x210/0x450
    [1459541.132273]  [<ffffffff816b00b9>] end_repeat_nmi+0x1e/0x7e
    [1459541.132570]  [<ffffffff810ec630>] ? __getnstimeofday64+0x90/0xd0
    [1459541.132888]  [<ffffffff810ec630>] ? __getnstimeofday64+0x90/0xd0
    [1459541.133207]  [<ffffffff810ec630>] ? __getnstimeofday64+0x90/0xd0
    [1459541.133518]  <<EOE>>  <IRQ>  [<ffffffff810ec67e>] getnstimeofday64+0xe/0x30    <---
    [1459541.133921]  [<ffffffff810ec735>] ktime_get_real+0x25/0x70
    [1459541.134214]  [<ffffffff81589246>] netif_receive_skb_internal+0x76/0xc0
    [1459541.134557]  [<ffffffff8158a318>] napi_gro_receive+0xd8/0x130
    [1459541.134912]  [<ffffffffc019c2c7>] igb_clean_rx_irq+0x387/0x6f0 [igb]
    [1459541.146843]  [<ffffffff8109d80a>] ? signal_wake_up_state+0x2a/0x30
    [1459541.158691]  [<ffffffffc019c9ab>] igb_poll+0x37b/0x760 [igb]
    [1459541.170351]  [<ffffffff810b5021>] ? run_posix_cpu_timers+0x51/0x830
    [1459541.182038]  [<ffffffff8158999d>] net_rx_action+0x16d/0x380
    [1459541.193630]  [<ffffffff810925bf>] __do_softirq+0xef/0x280
    [1459541.204942]  [<ffffffff816bd3ac>] call_softirq+0x1c/0x30
    [1459541.216073]  [<ffffffff8102d405>] do_softirq+0x65/0xa0
    [1459541.227106]  [<ffffffff81092945>] irq_exit+0x105/0x110
    [1459541.237865]  [<ffffffff816be986>] do_IRQ+0x56/0xe0
    [1459541.248444]  [<ffffffff816af4f2>] common_interrupt+0x232/0x232
    

Environment

  • Red Hat Enterprise Linux
    • 7.3 (kernel-3.10.0-514.el7 and above)
    • 7.4 (kernel-3.10.0-693.el7 and above)
    • 7.5 (kernel-3.10.0-862.el7 and above)
    • 7.6 (kernel-3.10.0-957.el7 and above)
    • 7.7 (kernel-3.10.0-1062.el7 and above)
    • 7.8 (kernel-3.10.0-1127.el7 and above)
    • 7.9 (kernel-3.10.0-1160.el7 and above)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content