Host panics when receiving flood of IO errors from storage target
Issue
-
While testing IO failover on the SAN devices connected to Qlogic HBAs, some of the FC remote ports were disabled. After this system got crashed with hard lockup and panic messages shown in below snip:
qla2xxx [0000:04:00.0]-801c:31: Abort command issued nexus=31:1:1 -- 1 2002. sd 31:0:1:1: tag#1 finish aborted command sd 25:0:1:1: tag#4 abort scheduled sd 164:0:1:1: tag#1 abort scheduled sd 164:0:1:1: tag#2 abort scheduled sd 193:0:1:1: tag#2 abort scheduled sd 193:0:1:1: tag#2 aborting command Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 [...] Call Trace: <NMI> [<ffffffff816351f1>] dump_stack+0x19/0x1b [<ffffffff8162ea6c>] panic+0xd8/0x1e7 [<ffffffff8111b450>] ? restart_watchdog_hrtimer+0x50/0x50 [<ffffffff8111b512>] watchdog_overflow_callback+0xc2/0xd0 [...]Below is the full back trace:
crash> bt PID: 0 TASK: ffffffff81951440 CPU: 0 COMMAND: "swapper/0" [...] #9 [ffff88042fa05ec8] do_nmi at ffffffff8163e450 #10 [ffff88042fa05ef0] end_repeat_nmi at ffffffff8163d771 [exception RIP: sys_imageblit+590] RIP: ffffffffa041c24e RSP: ffff88042fa03760 RFLAGS: 00000046 RAX: 0000000000000000 RBX: ffffc90013da6d7c RCX: 0000000000000000 RDX: 0000000000000050 RSI: 0000000000000000 RDI: 0000000000000001 RBP: ffff88042fa037d0 R8: ffffffffa041d060 R9: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880426474000 R13: ffff880827e6d625 R14: ffffc90013da6cc0 R15: ffff880827e6d620 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #11 [ffff88042fa03760] sys_imageblit at ffffffffa041c24e [sysimgblt] #12 [ffff88042fa037d8] mga_imageblit at ffffffffa03e761b [mgag200] #13 [ffff88042fa037f8] bit_putcs at ffffffff813565d7 #14 [ffff88042fa03910] fbcon_putcs at ffffffff813512e8 #15 [ffff88042fa03978] fbcon_redraw at ffffffff81351637 #16 [ffff88042fa039e8] fbcon_scroll at ffffffff81352ae6 #17 [ffff88042fa03a48] scrup at ffffffff813c3b6c #18 [ffff88042fa03a78] lf at ffffffff813c3c20 #19 [ffff88042fa03aa8] vt_console_print at ffffffff813c3f02 #20 [ffff88042fa03b10] call_console_drivers.constprop.15 at ffffffff8107c4d1 #21 [ffff88042fa03b38] console_unlock at ffffffff8107d6cf #22 [ffff88042fa03b70] vprintk_emit at ffffffff8107d9c6 #23 [ffff88042fa03be0] printk at ffffffff8162ec81 [...]
Environment
- Red Hat Enterprise Linux 7.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
