System with Emulex HBAs crashed with hard LOCKUP errors

Solution Verified - Updated -

Issue

  • A system with Emulex HBAs crashed with hard LOCKUP errors:

    sd 1:0:2:16: [sdad] tag#385 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
    sd 1:0:2:16: [sdad] tag#385 CDB: Test Unit Ready 00 00 00 00 00 00
    sd 1:0:2:17: [sdaf] tag#386 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
    sd 1:0:2:17: [sdaf] tag#386 CDB: Test Unit Ready 00 00 00 00 00 00
    [...]
    lpfc 0000:09:00.0: 0:2885 Port Status Event: port status reg 0x81800000, port smphr reg 0xc000, error 1=0x52004a02, error 2=0x0
    lpfc 0000:09:00.0: 0:2887 Reset Needed: Attempting Port Recovery...
    [...]
    NMI watchdog: Watchdog detected hard LOCKUP on cpu 21
    [...]
     CPU: 21 PID: 726 Comm: lpfc_worker_1 Kdump: loaded Tainted: P        W  OE  ------------ T 3.10.0-862.3.2.el7.x86_64 #1
     Hardware name: HP ProLiant BL460c Gen9, BIOS I36 10/25/2017
     Call Trace:
      <NMI>  [<ffffffff8270e78e>] dump_stack+0x19/0x1b
      [<ffffffff8213fd85>] watchdog_overflow_callback+0x135/0x140
      [<ffffffff8217f8b7>] __perf_event_overflow+0x57/0x100
      [<ffffffff821882a4>] perf_event_overflow+0x14/0x20
      [<ffffffff8200a580>] intel_pmu_handle_irq+0x220/0x510
      [<ffffffff8234cbb4>] ? ioremap_page_range+0x2b4/0x450
      [<ffffffff821d6c84>] ? vunmap_page_range+0x234/0x470
      [<ffffffff8240a946>] ? ghes_copy_tofrom_phys+0x116/0x210
      [<ffffffff8240aae0>] ? ghes_read_estatus+0xa0/0x190
      [<ffffffff82717031>] perf_event_nmi_handler+0x31/0x50
      [<ffffffff8271893c>] nmi_handle.isra.0+0x8c/0x150
      [<ffffffff82718c18>] do_nmi+0x218/0x460
      [<ffffffff82717dd3>] end_repeat_nmi+0x1e/0x81
      [<ffffffffc0cd4636>] ? lpfc_hba_clean_txcmplq+0x176/0x1f0 [lpfc]  <<----------
      [<ffffffffc0cd4636>] ? lpfc_hba_clean_txcmplq+0x176/0x1f0 [lpfc]  <<----------
      [<ffffffffc0cd4636>] ? lpfc_hba_clean_txcmplq+0x176/0x1f0 [lpfc]  <<----------
      <EOE>  [<ffffffffc0cd4c21>] lpfc_hba_down_post_s4+0x61/0x490 [lpfc]
      [<ffffffffc0cd5af2>] lpfc_hba_down_post+0x12/0x20 [lpfc]
      [<ffffffffc0c9fb61>] lpfc_sli_brdrestart_s4+0xe1/0x150 [lpfc]
      [<ffffffffc0c9fbe2>] lpfc_sli_brdrestart+0x12/0x20 [lpfc]
      [<ffffffffc0cda9d8>] lpfc_sli4_port_sta_fn_reset.constprop.36+0x48/0x150 [lpfc]
      [<ffffffffc0cdabde>] lpfc_handle_eratt_s4+0xfe/0x5a0 [lpfc]
      [<ffffffffc0cd6145>] lpfc_handle_eratt+0x15/0x20 [lpfc]
      [<ffffffffc0ccf5d0>] lpfc_work_done+0x11e0/0x14c0 [lpfc]
      [<ffffffff8271394c>] ? __schedule+0x41c/0xa20
      [<ffffffffc0ccf920>] lpfc_do_work+0x70/0x1e0 [lpfc]
      [<ffffffff820bc150>] ? wake_up_atomic_t+0x30/0x30
      [<ffffffffc0ccf8b0>] ? lpfc_work_done+0x14c0/0x14c0 [lpfc]
      [<ffffffff820bb161>] kthread+0xd1/0xe0
      [<ffffffff820bb090>] ? insert_kthread_work+0x40/0x40
      [<ffffffff82720677>] ret_from_fork_nospec_begin+0x21/0x21
      [<ffffffff820bb090>] ? insert_kthread_work+0x40/0x40
     Kernel panic - not syncing: Hard LOCKUP
    [...]
    

Environment

  • Red Hat Enterprise Linux 7.5
  • Emulex LPe32002 32G FC HBA
  • Emulex/HP LPe1605 16Gb FC HBA
  • kernel-3.10.0-855.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content