The system crashes due to hard lockup due to contention in system log at memcpy+0x5 or _spin_lock_irqsave+0x3f
Issue
The system log is full of failed I/O messages followed by a panic due to a hard lockup with the following call traces:
...many more I/O errors...
end_request: I/O error, dev sdab, sector 0
end_request: I/O error, dev sdm, sector 0
end_request: I/O error, dev sdv, sector 0
end_request: I/O error, dev sdg, sector 0
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
Pid: 35665, comm: udevadm Tainted: P 2.6.32-358.18.1.el6.x86_64 #1
Call Trace:
<NMI> [<ffffffff8150da18>] ? panic+0xa7/0x16f
[<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
[<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
[<ffffffff8101a89d>] ? x86_perf_event_update+0x5d/0xb0
[<ffffffff8101b82d>] ? x86_perf_event_set_period+0xdd/0x170
[<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
[<ffffffff810208c2>] ? intel_pmu_handle_irq+0x192/0x300
[<ffffffff815135f6>] ? kprobe_exceptions_notify+0x16/0x430
[<ffffffff81512169>] ? perf_event_nmi_handler+0x39/0xb0
[<ffffffff81513c25>] ? notifier_call_chain+0x55/0x80
[<ffffffff81513c8a>] ? atomic_notifier_call_chain+0x1a/0x20
[<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
[<ffffffff815118eb>] ? do_nmi+0x1bb/0x340
[<ffffffff815111b0>] ? nmi+0x20/0x30
[<ffffffff8151091f>] ? _spin_lock_irqsave+0x3f/0x40
<<EOE>> [<ffffffff812c24c7>] ? vgacon_cursor+0x77/0x250
[<ffffffff81336c59>] ? hide_cursor+0x29/0x90
[<ffffffff81338b5b>] ? vt_console_print+0x2fb/0x330
[<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
[<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
[<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
[<ffffffff8106f338>] ? vprintk+0x248/0x560
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
[<ffffffff8150db21>] ? printk+0x41/0x48
[<ffffffff812914ed>] ? pci_user_read_config_word+0x7d/0xc0
[<ffffffff812916ec>] ? pci_vpd_pci22_wait+0xfc/0x110
[<ffffffff81291902>] ? pci_vpd_pci22_read+0x92/0x180
[<ffffffff81290b5b>] ? pci_read_vpd+0x2b/0x30
[<ffffffff8129c9e0>] ? read_vpd_attr+0x30/0x40
[<ffffffff811fcd37>] ? read+0x127/0x210
[<ffffffff81181a95>] ? vfs_read+0xb5/0x1a0
[<ffffffff81181bd1>] ? sys_read+0x51/0x90
[<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Another possible call trace is:
...many more I/O errors...
end_request: I/O error, dev sdq, sector 232
end_request: I/O error, dev sdak, sector 112
end_request: I/O error, dev sdap, sector 80
end_request: I/O error, dev sdaa, sector 136
end_request: I/O error, dev sdc, sector 144
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 30
Pid: 5697, comm: dmp_daemon Tainted: P 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
<NMI> [<ffffffff8150daac>] ? panic+0xa7/0x16f
[<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
[<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
[<ffffffff8101a8ad>] ? x86_perf_event_update+0x5d/0xb0
[<ffffffff8101b83d>] ? x86_perf_event_set_period+0xdd/0x170
[<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
[<ffffffff810208d2>] ? intel_pmu_handle_irq+0x192/0x300
[<ffffffff81513686>] ? kprobe_exceptions_notify+0x16/0x430
[<ffffffff815121f9>] ? perf_event_nmi_handler+0x39/0xb0
[<ffffffff81513cb5>] ? notifier_call_chain+0x55/0x80
[<ffffffff81513d1a>] ? atomic_notifier_call_chain+0x1a/0x20
[<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
[<ffffffff8151197b>] ? do_nmi+0x1bb/0x340
[<ffffffff81511240>] ? nmi+0x20/0x30
[<ffffffff81283415>] ? memcpy+0x5/0x120
<<EOE>> [<ffffffff812c1ab8>] ? vgacon_scroll+0x298/0x2e0
[<ffffffff8127f62e>] ? number+0x2ee/0x320
[<ffffffff81336f40>] ? scrup+0xf0/0x100
[<ffffffff813370dd>] ? lf+0x5d/0x60
[<ffffffff81338a82>] ? vt_console_print+0x1d2/0x330
[<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
[<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
[<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
[<ffffffff8106f338>] ? vprintk+0x248/0x560
[<ffffffffa0628592>] ? dmp_notify_event+0x162/0x1e90 [vxdmp]
[<ffffffff8150dbb5>] ? printk+0x41/0x44
[<ffffffffa0629cf6>] ? dmp_notify_event+0x18c6/0x1e90 [vxdmp]
[<ffffffffa0609747>] ? dmp_return_io+0x87/0x1a0 [vxdmp]
[<ffffffffa0609944>] ? dmp_disk_failure+0xe4/0x110 [vxdmp]
[<ffffffffa06099d8>] ? dmp_error_action+0x68/0x170 [vxdmp]
[<ffffffffa062d11e>] ? dmp_error_analysis_callback+0xbe/0x130 [vxdmp]
[<ffffffffa06394c0>] ? dmp_process_scsireq+0x0/0x20 [vxdmp]
[<ffffffffa06394d2>] ? dmp_process_scsireq+0x12/0x20 [vxdmp]
[<ffffffffa063acd7>] ? dmp_daemons_loop+0x1d7/0x260 [vxdmp]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa063ab00>] ? dmp_daemons_loop+0x0/0x260 [vxdmp]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Environment
- Red Hat Enterprise Linux 6.4
- Red Hat Enterprise Linux 6.5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.