The kernel crashes due to a hard lockup occurred on a CPU. Not only the hard-locked up CPU but also other 9 CPUs are all looping in the same l3 list_lock spinlock.

Issue

• The kernel crashes due to a hard lockup occurred on a CPU.

Kernel panic - not syncing: Hard LOCKUP
Pid: 5703, comm: cmaidad Not tainted 2.6.32-754.18.2.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff81558d1a>] ? panic+0xa7/0x18b
 [<ffffffff8100fa35>] ? show_trace+0x15/0x20
 [<ffffffff810f7810>] ? watchdog_timer_fn+0x0/0x230
 [<ffffffff81130d2a>] ? __perf_event_overflow+0xaa/0x240
 [<ffffffff8101e908>] ? x86_perf_event_set_period+0xf8/0x180
 [<ffffffff81131384>] ? perf_event_overflow+0x14/0x20
 [<ffffffff81025e6c>] ? intel_pmu_handle_irq+0x21c/0x4a0
 [<ffffffff8155d0fb>] ? save_paranoid+0xfb/0x110
 [<ffffffff8155e86f>] ? perf_event_nmi_handler+0x3f/0xb0
 [<ffffffff81560380>] ? notifier_call_chain+0x50/0x80
 [<ffffffff8155d0ef>] ? save_paranoid+0xef/0x110
 [<ffffffff815603ea>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff810b14fe>] ? notify_die+0x2e/0x30
 [<ffffffff8155ded9>] ? do_nmi+0xd9/0x360
 [<ffffffff8155d7a4>] ? nmi+0x134/0x1a3
 [<ffffffff8155ca5e>] ? _spin_lock+0x1e/0x30
 <<EOE>>  [<ffffffff81186b5a>] ? cache_alloc_refill+0x8a/0x240
 [<ffffffff810134c9>] ? read_tsc+0x9/0x20
 [<ffffffff81187bfa>] ? kmem_cache_alloc+0x16a/0x1a0
 [<ffffffff811affb6>] ? getname+0x56/0x2a0
 [<ffffffff810f1283>] ? audit_filter_syscall+0x93/0xf0
 [<ffffffff8119e732>] ? do_sys_open+0x32/0x130
 [<ffffffff815642c2>] ? system_call_after_swapgs+0xa2/0x152
 [<ffffffff815642ce>] ? system_call_after_swapgs+0xae/0x152
 [<ffffffff815642c2>] ? system_call_after_swapgs+0xa2/0x152
 [<ffffffff8119e870>] ? sys_open+0x20/0x30
 [<ffffffff815643a7>] ? system_call_fastpath+0x35/0x3a
 [<ffffffff815642ce>] ? system_call_after_swapgs+0xae/0x152

• Not only the hard-locked-up CPU but also other 9 CPUs are all looping in the same l3 list_lock spinlock.

crash> bt -a | grep 'RIP: _spin_lock' -A3 | grep -oE 'RDI: ffff.{12}' | sort | uniq -c
WARNING: possibly bogus exception frame
     10 RDI: ffff88601f80ae80
      1 RDI: ffff88608081dbd0
      1 RDI: ffffffff81e54510