System crashed after dm-multipath path failures
Issue
-
The FCoE HBA present on system had encountered a fw hang after which paths to multipath devices were failed. Shortly after the path failures, system got rebooted with following panic messages:
[2321378.786537] qla2xxx [0000:81:00.7]-6006:4: Detect abort needed. [2321378.786539] qla2xxx [0000:81:00.7]-6007:4: Firmware hung. [2321378.787040] qla2xxx [0000:81:00.7]-b02f:4: HW State: NEED RESET [2321378.787045] qla2xxx [0000:81:00.7]-009b:4: Device state is 0x4 = Need Reset. [2321378.787047] qla2xxx [0000:81:00.7]-009d:4: Device state is 0x4 = Need Reset. [2321378.787050] qla2xxx [0000:81:00.7]-00af:4: Performing ISP error recovery - ha=ffff88befa5b4000. [2321378.787153] qla2xxx [0000:81:00.7]-00b4:4: Done chip reset cleanup. [2321378.794558] qlcnic 0000:81:00.0: Pause control frames disabled on all ports [2321378.794569] qlcnic 0000:81:00.0: firmware hang detected [2321378.794589] qlcnic 0000:81:00.1: Pause control frames disabled on all ports [2321378.794594] qlcnic 0000:81:00.1: firmware hang detected [2321378.794603] qlcnic 0000:81:00.1: Dumping hw/fw registers ... [2357691.366302] qla2xxx [0000:0b:00.7]-8040:2: PCI/Register disconnect, exiting ... [2357691.366303] qla2xxx [0000:0b:00.6]-015b:1: Disabling adapter. [2357691.366304] qla2xxx [0000:0b:00.7]-8041:2: PCI/Register disconnect, exiting ... [2357691.366308] qla2xxx [0000:0b:00.7]-015b:2: Disabling adapter. ... [2357701.370522] device-mapper: multipath: Failing path 70:208. [2357701.370523] device-mapper: multipath: Failing path 71:32. [2357701.373447] IP: [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60 [2357701.373814] PGD 0 [2357701.374164] Oops: 0002 [#1] SMP [2357701.377000] sd 2:0:3:12: alua: Detached [...] [2357701.380637] CPU: 7 PID: 4300 Comm: systemd-udevd Tainted: G OE ------------ 3.10.0-514.26.1.el7.x86_64 #1 [2357701.381401] Hardware name: LENOVO System x3650 M5: -[8871AC1]-/01GR174, BIOS -[TCE130J-2.40]- 04/11/2017 [2357701.382158] task: ffff88bee0954e70 ti: ffff889ff43cc000 task.ti: ffff889ff43cc000 [2357701.382918] RIP: 0010:[<ffffffff8168eb1f>] [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60 [2357701.383721] RSP: 0018:ffff889ff43cfd00 EFLAGS: 00010046 [...] [2357701.393097] Stack: [2357701.393955] ffff889ff43cfd38 ffffffff810b1757 ffff885ef6289000 0000000000000000 [2357701.394877] ffff885ef6289000 ffff88bef65aaa40 ffff885ef6289000 ffff889ff43cfd88 [2357701.395823] ffffffff81455d0d 0000000000000000 ffff88bee0954e70 ffffffff810b1b20 [2357701.396741] Call Trace: [2357701.397644] [<ffffffff810b1757>] prepare_to_wait+0x27/0x90 [2357701.398527] [<ffffffff81455d0d>] scsi_block_when_processing_errors+0xdd/0x140 [2357701.399403] [<ffffffff810b1b20>] ? wake_up_atomic_t+0x30/0x30 [2357701.400288] [<ffffffff8145398c>] scsi_nonblockable_ioctl+0xcc/0xf0 [2357701.401472] [<ffffffffa012697e>] sd_ioctl+0x6e/0x140 [sd_mod] [2357701.402782] [<ffffffff812fd1c2>] __blkdev_driver_ioctl+0x22/0x30 [2357701.404075] [<ffffffffa0068042>] dm_blk_ioctl+0x82/0x90 [dm_mod] [2357701.405362] [<ffffffff812fdb40>] blkdev_ioctl+0x270/0x980 [2357701.406663] [<ffffffff8121f15e>] ? mntput_no_expire+0x3e/0x120 [2357701.407986] [<ffffffff8123a7b1>] block_ioctl+0x41/0x50 [2357701.409310] [<ffffffff81212855>] do_vfs_ioctl+0x2d5/0x4b0 [2357701.410676] [<ffffffff81200b3e>] ? ____fput+0xe/0x10 [2357701.411992] [<ffffffff81212ad1>] SyS_ioctl+0xa1/0xc0 [2357701.413316] [<ffffffff81697809>] system_call_fastpath+0x16/0x1b [2357701.414686] Code: df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 0f 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 <f0> 0f c1 17 89 d1 c1 e9 10 66 39 d1 75 05 4c 89 c0 5d c3 83 e1 [2357701.417517] RIP [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60 [2357701.418886] RSP <ffff889ff43cfd00> [2357701.420205] CR2: 00000000000000a8
Environment
- Red Hat Enterprise Linux 7.3
- QLogic QLE3262 Converged Network Adapter
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.