While testing addition/deletion of the SAN devices, system crashed with error 'Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0'
Issue
- While trying to test the addition/deletion of the SAN devices using following steps the system crashed with error 'Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0':
Steps to Reproduce:
1. Boot on the 6.6 kernel
2. Inject faults on the fabric layer and disable/enable targets ports repeatedly
3. Keep rescanning the SAN devices using following script:
#!/bin/sh -ex
while true; do
date
rescan-scsi-bus.sh -r -a --hosts=7,8
rescan-scsi-bus.sh -a --hosts=7,8
sleep 1
done
3. System panics with following errors:
rport-7:0-3: blocked FC remote port time out: removing target and saving binding
scsi 7:0:5:0: Device offlined - not ready after error recovery
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
Pid: 603, comm: fc_wq_7 Not tainted 2.6.32-504.3.3.el6.x86_64 #1
Call Trace:
<NMI> [<ffffffff81529277>] ? panic+0xa7/0x16f
[<ffffffff81014959>] ? sched_clock+0x9/0x10
[<ffffffff810ea5dd>] ? watchdog_overflow_callback+0xcd/0xd0
[<ffffffff81120d87>] ? __perf_event_overflow+0xa7/0x240
[<ffffffff8101dacd>] ? x86_perf_event_set_period+0xdd/0x170
[<ffffffff811213d4>] ? perf_event_overflow+0x14/0x20
[<ffffffff81024892>] ? intel_pmu_handle_irq+0x202/0x410
[<ffffffff8152e579>] ? perf_event_nmi_handler+0x39/0xb0
[<ffffffff81530035>] ? notifier_call_chain+0x55/0x80
[<ffffffff8153009a>] ? atomic_notifier_call_chain+0x1a/0x20
[<ffffffff810a4eae>] ? notify_die+0x2e/0x30
[<ffffffff8152dcfb>] ? do_nmi+0x1bb/0x340
[<ffffffff8152d5c0>] ? nmi+0x20/0x30
[<ffffffff8138ddf3>] ? scsi_remove_target+0x93/0x210
<<EOE>> [<ffffffffa0235570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[<ffffffffa0235596>] ? fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[<ffffffff81097fe0>] ? worker_thread+0x170/0x2a0
[<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81097e70>] ? worker_thread+0x0/0x2a0
[<ffffffff8109e66e>] ? kthread+0x9e/0xc0
[<ffffffff8100c20a>] ? child_rip+0xa/0x20
[<ffffffff8109e5d0>] ? kthread+0x0/0xc0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Environment
- Red Hat Enterprise Linux 6.6
- kernel 2.6.32-504.3.3.el6.x86_64
- kernel-2.6.32-504.8.1.el6.x86_64
- Fiber Channel HBA
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.