While testing addition/deletion of the SAN devices, system crashed with error 'Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0'

Solution Verified - Updated -

Issue

  • While trying to test the addition/deletion of the SAN devices using following steps the system crashed with error 'Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0':

Steps to Reproduce:
1. Boot on the 6.6 kernel
2. Inject faults on the fabric layer and disable/enable targets ports repeatedly
3. Keep rescanning the SAN devices using following script:

    #!/bin/sh -ex
    while true; do
    date
    rescan-scsi-bus.sh -r -a --hosts=7,8
    rescan-scsi-bus.sh -a --hosts=7,8
    sleep 1
    done

3. System panics with following errors:

      rport-7:0-3: blocked FC remote port time out: removing target and saving binding
     scsi 7:0:5:0: Device offlined - not ready after error recovery
     Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
     Pid: 603, comm: fc_wq_7 Not tainted 2.6.32-504.3.3.el6.x86_64 #1
     Call Trace:
      <NMI>  [<ffffffff81529277>] ? panic+0xa7/0x16f
      [<ffffffff81014959>] ? sched_clock+0x9/0x10
      [<ffffffff810ea5dd>] ? watchdog_overflow_callback+0xcd/0xd0
      [<ffffffff81120d87>] ? __perf_event_overflow+0xa7/0x240
      [<ffffffff8101dacd>] ? x86_perf_event_set_period+0xdd/0x170
      [<ffffffff811213d4>] ? perf_event_overflow+0x14/0x20
      [<ffffffff81024892>] ? intel_pmu_handle_irq+0x202/0x410
      [<ffffffff8152e579>] ? perf_event_nmi_handler+0x39/0xb0
      [<ffffffff81530035>] ? notifier_call_chain+0x55/0x80
      [<ffffffff8153009a>] ? atomic_notifier_call_chain+0x1a/0x20
      [<ffffffff810a4eae>] ? notify_die+0x2e/0x30
      [<ffffffff8152dcfb>] ? do_nmi+0x1bb/0x340
      [<ffffffff8152d5c0>] ? nmi+0x20/0x30
      [<ffffffff8138ddf3>] ? scsi_remove_target+0x93/0x210
      <<EOE>>  [<ffffffffa0235570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
      [<ffffffffa0235596>] ? fc_starget_delete+0x26/0x30 [scsi_transport_fc]
      [<ffffffff81097fe0>] ? worker_thread+0x170/0x2a0
      [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
      [<ffffffff81097e70>] ? worker_thread+0x0/0x2a0
      [<ffffffff8109e66e>] ? kthread+0x9e/0xc0
      [<ffffffff8100c20a>] ? child_rip+0xa/0x20
      [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
      [<ffffffff8100c200>] ? child_rip+0x0/0x20

Environment

  • Red Hat Enterprise Linux 6.6
  • kernel 2.6.32-504.3.3.el6.x86_64
  • kernel-2.6.32-504.8.1.el6.x86_64
  • Fiber Channel HBA

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.