The system crashes due to hard lockup due to contention in system log at memcpy+0x5 or _spin_lock_irqsave+0x3f

Solution Unverified - Updated -

Issue

The system log is full of failed I/O messages followed by a panic due to a hard lockup with the following call traces:

...many more I/O errors...
end_request: I/O error, dev sdab, sector 0
end_request: I/O error, dev sdm, sector 0
end_request: I/O error, dev sdv, sector 0
end_request: I/O error, dev sdg, sector 0
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
Pid: 35665, comm: udevadm Tainted: P 2.6.32-358.18.1.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff8150da18>] ? panic+0xa7/0x16f
 [<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
 [<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
 [<ffffffff8101a89d>] ? x86_perf_event_update+0x5d/0xb0
 [<ffffffff8101b82d>] ? x86_perf_event_set_period+0xdd/0x170
 [<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
 [<ffffffff810208c2>] ? intel_pmu_handle_irq+0x192/0x300
 [<ffffffff815135f6>] ? kprobe_exceptions_notify+0x16/0x430
 [<ffffffff81512169>] ? perf_event_nmi_handler+0x39/0xb0
 [<ffffffff81513c25>] ? notifier_call_chain+0x55/0x80
 [<ffffffff81513c8a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
 [<ffffffff815118eb>] ? do_nmi+0x1bb/0x340
 [<ffffffff815111b0>] ? nmi+0x20/0x30
 [<ffffffff8151091f>] ? _spin_lock_irqsave+0x3f/0x40
 <<EOE>>  [<ffffffff812c24c7>] ? vgacon_cursor+0x77/0x250
 [<ffffffff81336c59>] ? hide_cursor+0x29/0x90
 [<ffffffff81338b5b>] ? vt_console_print+0x2fb/0x330
 [<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
 [<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
 [<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
 [<ffffffff8106f338>] ? vprintk+0x248/0x560
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8150db21>] ? printk+0x41/0x48
 [<ffffffff812914ed>] ? pci_user_read_config_word+0x7d/0xc0
 [<ffffffff812916ec>] ? pci_vpd_pci22_wait+0xfc/0x110
 [<ffffffff81291902>] ? pci_vpd_pci22_read+0x92/0x180
 [<ffffffff81290b5b>] ? pci_read_vpd+0x2b/0x30
 [<ffffffff8129c9e0>] ? read_vpd_attr+0x30/0x40
 [<ffffffff811fcd37>] ? read+0x127/0x210
 [<ffffffff81181a95>] ? vfs_read+0xb5/0x1a0
 [<ffffffff81181bd1>] ? sys_read+0x51/0x90
 [<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

Another possible call trace is:

...many more I/O errors...
end_request: I/O error, dev sdq, sector 232
end_request: I/O error, dev sdak, sector 112
end_request: I/O error, dev sdap, sector 80
end_request: I/O error, dev sdaa, sector 136
end_request: I/O error, dev sdc, sector 144
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 30
Pid: 5697, comm: dmp_daemon Tainted: P 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff8150daac>] ? panic+0xa7/0x16f
 [<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
 [<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
 [<ffffffff8101a8ad>] ? x86_perf_event_update+0x5d/0xb0
 [<ffffffff8101b83d>] ? x86_perf_event_set_period+0xdd/0x170
 [<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
 [<ffffffff810208d2>] ? intel_pmu_handle_irq+0x192/0x300
 [<ffffffff81513686>] ? kprobe_exceptions_notify+0x16/0x430
 [<ffffffff815121f9>] ? perf_event_nmi_handler+0x39/0xb0
 [<ffffffff81513cb5>] ? notifier_call_chain+0x55/0x80
 [<ffffffff81513d1a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
 [<ffffffff8151197b>] ? do_nmi+0x1bb/0x340
 [<ffffffff81511240>] ? nmi+0x20/0x30
 [<ffffffff81283415>] ? memcpy+0x5/0x120
 <<EOE>>  [<ffffffff812c1ab8>] ? vgacon_scroll+0x298/0x2e0
 [<ffffffff8127f62e>] ? number+0x2ee/0x320
 [<ffffffff81336f40>] ? scrup+0xf0/0x100
 [<ffffffff813370dd>] ? lf+0x5d/0x60
 [<ffffffff81338a82>] ? vt_console_print+0x1d2/0x330
 [<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
 [<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
 [<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
 [<ffffffff8106f338>] ? vprintk+0x248/0x560
 [<ffffffffa0628592>] ? dmp_notify_event+0x162/0x1e90 [vxdmp]
 [<ffffffff8150dbb5>] ? printk+0x41/0x44
 [<ffffffffa0629cf6>] ? dmp_notify_event+0x18c6/0x1e90 [vxdmp]
 [<ffffffffa0609747>] ? dmp_return_io+0x87/0x1a0 [vxdmp]
 [<ffffffffa0609944>] ? dmp_disk_failure+0xe4/0x110 [vxdmp]
 [<ffffffffa06099d8>] ? dmp_error_action+0x68/0x170 [vxdmp]
 [<ffffffffa062d11e>] ? dmp_error_analysis_callback+0xbe/0x130 [vxdmp]
 [<ffffffffa06394c0>] ? dmp_process_scsireq+0x0/0x20 [vxdmp]
 [<ffffffffa06394d2>] ? dmp_process_scsireq+0x12/0x20 [vxdmp]
 [<ffffffffa063acd7>] ? dmp_daemons_loop+0x1d7/0x260 [vxdmp]
 [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
 [<ffffffffa063ab00>] ? dmp_daemons_loop+0x0/0x260 [vxdmp]
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

Environment

  • Red Hat Enterprise Linux 6.4
  • Red Hat Enterprise Linux 6.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content