Host panics when receiving flood of IO errors from storage target

Solution Verified - Updated -

Issue

  • While testing IO failover on the SAN devices connected to Qlogic HBAs, some of the FC remote ports were disabled. After this system got crashed with hard lockup and panic messages shown in below snip:

    qla2xxx [0000:04:00.0]-801c:31: Abort command issued nexus=31:1:1 --  1 2002.
    sd 31:0:1:1: tag#1 finish aborted command
    sd 25:0:1:1: tag#4 abort scheduled
    sd 164:0:1:1: tag#1 abort scheduled
    sd 164:0:1:1: tag#2 abort scheduled
    sd 193:0:1:1: tag#2 abort scheduled
    sd 193:0:1:1: tag#2 aborting command
    Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
    [...]
    Call Trace:
     <NMI>  [<ffffffff816351f1>] dump_stack+0x19/0x1b
     [<ffffffff8162ea6c>] panic+0xd8/0x1e7
     [<ffffffff8111b450>] ? restart_watchdog_hrtimer+0x50/0x50
     [<ffffffff8111b512>] watchdog_overflow_callback+0xc2/0xd0
    [...]
    

    Below is the full back trace:

    crash> bt
    PID: 0      TASK: ffffffff81951440  CPU: 0   COMMAND: "swapper/0"
    [...]
     #9 [ffff88042fa05ec8] do_nmi at ffffffff8163e450
    #10 [ffff88042fa05ef0] end_repeat_nmi at ffffffff8163d771
        [exception RIP: sys_imageblit+590]
        RIP: ffffffffa041c24e  RSP: ffff88042fa03760  RFLAGS: 00000046
        RAX: 0000000000000000  RBX: ffffc90013da6d7c  RCX: 0000000000000000
        RDX: 0000000000000050  RSI: 0000000000000000  RDI: 0000000000000001
        RBP: ffff88042fa037d0   R8: ffffffffa041d060   R9: 0000000000000000
        R10: 0000000000000001  R11: 0000000000000000  R12: ffff880426474000
        R13: ffff880827e6d625  R14: ffffc90013da6cc0  R15: ffff880827e6d620
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    --- <NMI exception stack> ---
    #11 [ffff88042fa03760] sys_imageblit at ffffffffa041c24e [sysimgblt]
    #12 [ffff88042fa037d8] mga_imageblit at ffffffffa03e761b [mgag200]
    #13 [ffff88042fa037f8] bit_putcs at ffffffff813565d7
    #14 [ffff88042fa03910] fbcon_putcs at ffffffff813512e8
    #15 [ffff88042fa03978] fbcon_redraw at ffffffff81351637
    #16 [ffff88042fa039e8] fbcon_scroll at ffffffff81352ae6
    #17 [ffff88042fa03a48] scrup at ffffffff813c3b6c
    #18 [ffff88042fa03a78] lf at ffffffff813c3c20
    #19 [ffff88042fa03aa8] vt_console_print at ffffffff813c3f02
    #20 [ffff88042fa03b10] call_console_drivers.constprop.15 at ffffffff8107c4d1
    #21 [ffff88042fa03b38] console_unlock at ffffffff8107d6cf
    #22 [ffff88042fa03b70] vprintk_emit at ffffffff8107d9c6
    #23 [ffff88042fa03be0] printk at ffffffff8162ec81
    [...]
    

Environment

  • Red Hat Enterprise Linux 7.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.