Issue

The system log is full of failed I/O messages followed by a panic due to a hard lockup with the following call traces:

...many more I/O errors...
end_request: I/O error, dev sdab, sector 0
end_request: I/O error, dev sdm, sector 0
end_request: I/O error, dev sdv, sector 0
end_request: I/O error, dev sdg, sector 0
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
Pid: 35665, comm: udevadm Tainted: P 2.6.32-358.18.1.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff8150da18>] ? panic+0xa7/0x16f
 [<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
 [<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
 [<ffffffff8101a89d>] ? x86_perf_event_update+0x5d/0xb0
 [<ffffffff8101b82d>] ? x86_perf_event_set_period+0xdd/0x170
 [<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
 [<ffffffff810208c2>] ? intel_pmu_handle_irq+0x192/0x300
 [<ffffffff815135f6>] ? kprobe_exceptions_notify+0x16/0x430
 [<ffffffff81512169>] ? perf_event_nmi_handler+0x39/0xb0
 [<ffffffff81513c25>] ? notifier_call_chain+0x55/0x80
 [<ffffffff81513c8a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
 [<ffffffff815118eb>] ? do_nmi+0x1bb/0x340
 [<ffffffff815111b0>] ? nmi+0x20/0x30
 [<ffffffff8151091f>] ? _spin_lock_irqsave+0x3f/0x40
 <<EOE>>  [<ffffffff812c24c7>] ? vgacon_cursor+0x77/0x250
 [<ffffffff81336c59>] ? hide_cursor+0x29/0x90
 [<ffffffff81338b5b>] ? vt_console_print+0x2fb/0x330
 [<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
 [<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
 [<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
 [<ffffffff8106f338>] ? vprintk+0x248/0x560
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8150db21>] ? printk+0x41/0x48
 [<ffffffff812914ed>] ? pci_user_read_config_word+0x7d/0xc0
 [<ffffffff812916ec>] ? pci_vpd_pci22_wait+0xfc/0x110
 [<ffffffff81291902>] ? pci_vpd_pci22_read+0x92/0x180
 [<ffffffff81290b5b>] ? pci_read_vpd+0x2b/0x30
 [<ffffffff8129c9e0>] ? read_vpd_attr+0x30/0x40
 [<ffffffff811fcd37>] ? read+0x127/0x210
 [<ffffffff81181a95>] ? vfs_read+0xb5/0x1a0
 [<ffffffff81181bd1>] ? sys_read+0x51/0x90
 [<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

Another possible call trace is:

...many more I/O errors...
end_request: I/O error, dev sdq, sector 232
end_request: I/O error, dev sdak, sector 112
end_request: I/O error, dev sdap, sector 80
end_request: I/O error, dev sdaa, sector 136
end_request: I/O error, dev sdc, sector 144
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 30
Pid: 5697, comm: dmp_daemon Tainted: P 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
 <NMI>  [<ffffffff8150daac>] ? panic+0xa7/0x16f
 [<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
 [<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
 [<ffffffff8101a8ad>] ? x86_perf_event_update+0x5d/0xb0
 [<ffffffff8101b83d>] ? x86_perf_event_set_period+0xdd/0x170
 [<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
 [<ffffffff810208d2>] ? intel_pmu_handle_irq+0x192/0x300
 [<ffffffff81513686>] ? kprobe_exceptions_notify+0x16/0x430
 [<ffffffff815121f9>] ? perf_event_nmi_handler+0x39/0xb0
 [<ffffffff81513cb5>] ? notifier_call_chain+0x55/0x80
 [<ffffffff81513d1a>] ? atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
 [<ffffffff8151197b>] ? do_nmi+0x1bb/0x340
 [<ffffffff81511240>] ? nmi+0x20/0x30
 [<ffffffff81283415>] ? memcpy+0x5/0x120
 <<EOE>>  [<ffffffff812c1ab8>] ? vgacon_scroll+0x298/0x2e0
 [<ffffffff8127f62e>] ? number+0x2ee/0x320
 [<ffffffff81336f40>] ? scrup+0xf0/0x100
 [<ffffffff813370dd>] ? lf+0x5d/0x60
 [<ffffffff81338a82>] ? vt_console_print+0x1d2/0x330
 [<ffffffff8106e585>] ? __call_console_drivers+0x75/0x90
 [<ffffffff8106e5ea>] ? _call_console_drivers+0x4a/0x80
 [<ffffffff8106ec38>] ? release_console_sem+0x108/0x220
 [<ffffffff8106f338>] ? vprintk+0x248/0x560
 [<ffffffffa0628592>] ? dmp_notify_event+0x162/0x1e90 [vxdmp]
 [<ffffffff8150dbb5>] ? printk+0x41/0x44
 [<ffffffffa0629cf6>] ? dmp_notify_event+0x18c6/0x1e90 [vxdmp]
 [<ffffffffa0609747>] ? dmp_return_io+0x87/0x1a0 [vxdmp]
 [<ffffffffa0609944>] ? dmp_disk_failure+0xe4/0x110 [vxdmp]
 [<ffffffffa06099d8>] ? dmp_error_action+0x68/0x170 [vxdmp]
 [<ffffffffa062d11e>] ? dmp_error_analysis_callback+0xbe/0x130 [vxdmp]
 [<ffffffffa06394c0>] ? dmp_process_scsireq+0x0/0x20 [vxdmp]
 [<ffffffffa06394d2>] ? dmp_process_scsireq+0x12/0x20 [vxdmp]
 [<ffffffffa063acd7>] ? dmp_daemons_loop+0x1d7/0x260 [vxdmp]
 [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
 [<ffffffffa063ab00>] ? dmp_daemons_loop+0x0/0x260 [vxdmp]
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20