A VM crashes due to a hard LOCKUP where one task is iterating over spinlock busy-wait in get_partial_node()

Solution Unverified - Updated -

Issue

  • A VM crashed due to a hard LOCKUP
[Wed Oct  9 07:34:16 EDT 2024] Kernel panic - not syncing: Hard LOCKUP
[Wed Oct  9 07:34:16 EDT 2024] CPU: 9 PID: 75990 Comm: evdefer/4 Kdump: loaded Tainted: P            E  ------------   3.10.0-1160.36.2.el7.x86_64 #1
[Wed Oct  9 07:34:16 EDT 2024] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[Wed Oct  9 07:34:16 EDT 2024] Call Trace:
[Wed Oct  9 07:34:16 EDT 2024]  <NMI>  [<ffffffff82183559>] dump_stack+0x19/0x1b
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8217d261>] panic+0xe8/0x21f
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81a2f958>] ? show_regs+0x58/0x290
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81a9af3f>] nmi_panic+0x3f/0x40
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81b4f199>] watchdog_overflow_callback+0x119/0x140
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81ba8d57>] __perf_event_overflow+0x57/0x100
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81bb2554>] perf_event_overflow+0x14/0x20
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81a05595>] x86_pmu_handle_irq+0x125/0x180
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81a066f5>] amd_pmu_handle_irq+0x35/0x80
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8218c031>] perf_event_nmi_handler+0x31/0x50
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8218d93c>] nmi_handle.isra.0+0x8c/0x150
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8218dc18>] do_nmi+0x218/0x460
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8218cd9c>] end_repeat_nmi+0x1e/0x81
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81b1797d>] ? native_queued_spin_lock_slowpath+0x1d/0x200
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81b1797d>] ? native_queued_spin_lock_slowpath+0x1d/0x200
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81b1797d>] ? native_queued_spin_lock_slowpath+0x1d/0x200
[Wed Oct  9 07:34:16 EDT 2024]  <EOE>  [<ffffffff8217dd13>] queued_spin_lock_slowpath+0xb/0xf
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8218bac0>] _raw_spin_lock+0x20/0x30
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c2657b>] get_partial_node.isra.48+0x4b/0x1f0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c3ece2>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81bce09e>] ? lru_cache_add+0xe/0x10
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81bce09e>] ? lru_cache_add+0xe/0x10
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc08247ce>] ? 0xffffffffc08247cd
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81bf675d>] ? handle_mm_fault+0xbfd/0xfb0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc08247ce>] ? 0xffffffffc08247cd
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff8217fe85>] __slab_alloc+0x40/0x5c
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c28c90>] __kmalloc+0x1c0/0x230
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc08247ce>] 0xffffffffc08247cd
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc09b996e>] cshook_systemcalltable_pre_compat_sys_ioctl+0x3471e/0x3d3e0 [falcon_lsm_serviceable]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc097717d>] cshook_network_ops_inet6_sockraw_release+0x20d7d/0x21120 [falcon_lsm_serviceable]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc0977923>] cshook_security_file_permission+0x103/0x1140 [falcon_lsm_serviceable]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc09c27ce>] cshook_security_file_open+0xe/0x10 [falcon_lsm_serviceable]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc08908b3>] pinnedhook_security_file_open+0x43/0x70 [falcon_lsm_pinned_17005]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81d09502>] security_file_open+0x22/0x70
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c4bb49>] do_dentry_open+0xc9/0x2d0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81d08fa2>] ? security_inode_permission+0x22/0x30
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c4bdea>] vfs_open+0x5a/0xb0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c5a2aa>] ? may_open+0x5a/0x120
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c5e1d6>] do_last+0x1f6/0x1340
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c5f3ed>] path_openat+0xcd/0x5a0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc04fff65>] ? xfs_file_buffered_aio_read+0xa5/0x110 [xfs]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c6163d>] do_filp_open+0x4d/0xb0
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c6f737>] ? __alloc_fd+0x47/0x170
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c4d3d4>] do_sys_open+0x124/0x220
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffff81c4d4ee>] SyS_open+0x1e/0x20
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc078a4ac>] unload_network_ops_symbols+0x9dac/0xa380 [falcon_lsm_pinned_16004]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc07b74ac>] unload_network_ops_symbols+0x9dac/0xa380 [falcon_lsm_pinned_16108]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc07d54ac>] unload_network_ops_symbols+0x9dac/0xa380 [falcon_lsm_pinned_16206]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc07f376c>] unload_network_ops_symbols+0x9dac/0xa380 [falcon_lsm_pinned_16407]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc081176c>] unload_network_ops_symbols+0x9dac/0xa380 [falcon_lsm_pinned_16604]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc084076c>] unload_network_ops_symbols+0xadac/0xb390 [falcon_lsm_pinned_16703]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc085f77c>] unload_network_ops_symbols+0xadac/0xb390 [falcon_lsm_pinned_16803]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc087e77c>] unload_network_ops_symbols+0xadac/0xb390 [falcon_lsm_pinned_16903]
[Wed Oct  9 07:34:16 EDT 2024]  [<ffffffffc089d77c>] unload_network_ops_symbols+0xadac/0xb390 [falcon_lsm_pinned_17005]

Environment

  • Red Hat Enterprise Linux 7.9.z
  • VMware Virtual Machine

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content