NMI Watchdog detected hard lockup on the CPU running user-space code

Solution Unverified - Updated -

Issue

  • NMI Watchdog detected hard lockup on the CPU running user-space code, and the system crashed with the following call traces:
[Fri May  6 09:43:23 EDT 2022] NMI watchdog: Watchdog detected hard LOCKUP on cpu 21
[Fri May  6 09:43:23 EDT 2022] Modules linked in: iptable_filter ...
[Fri May  6 09:43:23 EDT 2022] libata tg3 megaraid_sas mdio mtd drm_panel_orientation_quirks ptp pps_core nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
[Fri May  6 09:43:23 EDT 2022] CPU: 21 PID: 239601 Comm: kepler_data_bor Kdump: loaded Tainted: GOE  ------------  3.10.0-1160.21.1.el7.x86_64 #1
[Fri May  6 09:43:23 EDT 2022] Hardware name: Lenovo Thinksystem SR665/7D2VCTO1WW, BIOS D8E110K-1.21 12/16/2020
[Fri May  6 09:43:23 EDT 2022] task: ffff950894392100 ti: ffff9508fdf98000 task.ti: ffff9508fdf98000
[Fri May  6 09:43:23 EDT 2022] RIP: 0033:[<00007fdd0a532832>]  [<00007fdd0a532832>] 0x7fdd0a532832
[Fri May  6 09:43:23 EDT 2022] RSP: 002b:00007fff7cc07750  EFLAGS: 00000246
[Fri May  6 09:43:23 EDT 2022] RAX: 00007fdd0a83fba0 RBX: 000056435e47b8b0 RCX: 7fffffffffffffff
[Fri May  6 09:43:23 EDT 2022] RDX: 0000000000000000 RSI: 00007fdcedcbb510 RDI: 00007fdcf57678f0
[Fri May  6 09:43:23 EDT 2022] RBP: 00007fdcf57678f0 R08: 00000000ffffffff R09: 00007fff7cc07480
[Fri May  6 09:43:23 EDT 2022] R10: 0000000000000000 R11: 0000000000000001 R12: 00007fdd0aa546c0
[Fri May  6 09:43:23 EDT 2022] R13: 000056435e47bad0 R14: 00007fdc7f18f680 R15: 00007fdcb7522e50
[Fri May  6 09:43:23 EDT 2022] FS:  00007fdd0aa8d740(0000) GS:ffff96024e540000(0000) knlGS:0000000000000000
[Fri May  6 09:43:23 EDT 2022] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri May  6 09:43:23 EDT 2022] CR2: 00007fdc8de6d010 CR3: 000001079bbb2000 CR4: 0000000000340fe0
[Fri May  6 09:43:23 EDT 2022] 
[Fri May  6 09:43:23 EDT 2022] Kernel panic - not syncing: Hard LOCKUP
[Fri May  6 09:43:23 EDT 2022] CPU: 21 PID: 239601 Comm: kepler_data_bor Kdump: loaded Tainted: GOE  ------------  3.10.0-1160.21.1.el7.x86_64 #1
[Fri May  6 09:43:23 EDT 2022] Hardware name: Lenovo Thinksystem SR665/7D2VCTO1WW, BIOS D8E110K-1.21 12/16/2020
[Fri May  6 09:43:23 EDT 2022] Call Trace:
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb998305a>] dump_stack+0x19/0x1b
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb997c5b2>] panic+0xe8/0x21f
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb922f958>] ? show_regs+0x58/0x290
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb929ae9f>] nmi_panic+0x3f/0x40
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb934eeb9>] watchdog_overflow_callback+0x119/0x140
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb93a8a47>] __perf_event_overflow+0x57/0x100
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb93b2244>] perf_event_overflow+0x14/0x20
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb9205595>] x86_pmu_handle_irq+0x125/0x180
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb9337df1>] ? audit_copy_inode+0xa1/0xd0
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb92e2ffb>] ? should_numa_migrate_memory+0x5b/0x150
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb93f34be>] ? do_numa_page+0x1be/0x250
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb93f5e0b>] ? handle_mm_fault+0xadb/0xfb0
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb9297bfa>] ? get_task_exe_file+0x3a/0x50
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb9340853>] ? audit_exe_compare+0x43/0x50
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb92066f5>] amd_pmu_handle_irq+0x35/0x80
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb998c031>] perf_event_nmi_handler+0x31/0x50
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb998d93c>] nmi_handle.isra.0+0x8c/0x150
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb998db5d>] do_nmi+0x15d/0x460
[Fri May  6 09:43:23 EDT 2022]  [<ffffffffb998ccc9>] nmi+0x109/0x158

Environment

  • Red Hat Enterprise Linux 7
  • Lenovo Thinksystem SR665
  • AMD EPYC 7542 32-Core Processor

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content