System crashed after dm-multipath path failures

Solution In Progress - Updated -

Issue

  • The FCoE HBA present on system had encountered a fw hang after which paths to multipath devices were failed. Shortly after the path failures, system got rebooted with following panic messages:

    [2321378.786537] qla2xxx [0000:81:00.7]-6006:4: Detect abort  needed.
    [2321378.786539] qla2xxx [0000:81:00.7]-6007:4: Firmware hung.
    [2321378.787040] qla2xxx [0000:81:00.7]-b02f:4: HW State: NEED RESET
    [2321378.787045] qla2xxx [0000:81:00.7]-009b:4: Device state is 0x4 = Need Reset.
    [2321378.787047] qla2xxx [0000:81:00.7]-009d:4: Device state is 0x4 = Need Reset.
    [2321378.787050] qla2xxx [0000:81:00.7]-00af:4: Performing ISP error recovery - ha=ffff88befa5b4000.
    [2321378.787153] qla2xxx [0000:81:00.7]-00b4:4: Done chip reset cleanup.
    [2321378.794558] qlcnic 0000:81:00.0: Pause control frames disabled on all ports
    [2321378.794569] qlcnic 0000:81:00.0: firmware hang detected
    [2321378.794589] qlcnic 0000:81:00.1: Pause control frames disabled on all ports
    [2321378.794594] qlcnic 0000:81:00.1: firmware hang detected
    [2321378.794603] qlcnic 0000:81:00.1: Dumping hw/fw registers
    ...
    [2357691.366302] qla2xxx [0000:0b:00.7]-8040:2: PCI/Register disconnect, exiting
    ...
    [2357691.366303] qla2xxx [0000:0b:00.6]-015b:1: Disabling adapter.
    [2357691.366304] qla2xxx [0000:0b:00.7]-8041:2: PCI/Register disconnect, exiting
    ...
    [2357691.366308] qla2xxx [0000:0b:00.7]-015b:2: Disabling adapter.
    ...
    [2357701.370522] device-mapper: multipath: Failing path 70:208.
    [2357701.370523] device-mapper: multipath: Failing path 71:32.
    [2357701.373447] IP: [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.373814] PGD 0 
    [2357701.374164] Oops: 0002 [#1] SMP 
    [2357701.377000] sd 2:0:3:12: alua: Detached
    [...]
    [2357701.380637] CPU: 7 PID: 4300 Comm: systemd-udevd Tainted: G           OE  ------------   3.10.0-514.26.1.el7.x86_64 #1
    [2357701.381401] Hardware name: LENOVO System x3650 M5: -[8871AC1]-/01GR174, BIOS -[TCE130J-2.40]- 04/11/2017
    [2357701.382158] task: ffff88bee0954e70 ti: ffff889ff43cc000 task.ti: ffff889ff43cc000
    [2357701.382918] RIP: 0010:[<ffffffff8168eb1f>]  [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.383721] RSP: 0018:ffff889ff43cfd00  EFLAGS: 00010046
    [...]
    [2357701.393097] Stack:
    [2357701.393955]  ffff889ff43cfd38 ffffffff810b1757 ffff885ef6289000 0000000000000000
    [2357701.394877]  ffff885ef6289000 ffff88bef65aaa40 ffff885ef6289000 ffff889ff43cfd88
    [2357701.395823]  ffffffff81455d0d 0000000000000000 ffff88bee0954e70 ffffffff810b1b20
    [2357701.396741] Call Trace:
    [2357701.397644]  [<ffffffff810b1757>] prepare_to_wait+0x27/0x90
    [2357701.398527]  [<ffffffff81455d0d>] scsi_block_when_processing_errors+0xdd/0x140
    [2357701.399403]  [<ffffffff810b1b20>] ? wake_up_atomic_t+0x30/0x30
    [2357701.400288]  [<ffffffff8145398c>] scsi_nonblockable_ioctl+0xcc/0xf0
    [2357701.401472]  [<ffffffffa012697e>] sd_ioctl+0x6e/0x140 [sd_mod]
    [2357701.402782]  [<ffffffff812fd1c2>] __blkdev_driver_ioctl+0x22/0x30
    [2357701.404075]  [<ffffffffa0068042>] dm_blk_ioctl+0x82/0x90 [dm_mod]
    [2357701.405362]  [<ffffffff812fdb40>] blkdev_ioctl+0x270/0x980
    [2357701.406663]  [<ffffffff8121f15e>] ? mntput_no_expire+0x3e/0x120
    [2357701.407986]  [<ffffffff8123a7b1>] block_ioctl+0x41/0x50
    [2357701.409310]  [<ffffffff81212855>] do_vfs_ioctl+0x2d5/0x4b0
    [2357701.410676]  [<ffffffff81200b3e>] ? ____fput+0xe/0x10
    [2357701.411992]  [<ffffffff81212ad1>] SyS_ioctl+0xa1/0xc0
    [2357701.413316]  [<ffffffff81697809>] system_call_fastpath+0x16/0x1b
    [2357701.414686] Code: df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 0f 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 <f0> 0f c1 17 89 d1 c1 e9 10 66 39 d1 75 05 4c 89 c0 5d c3 83 e1 
    [2357701.417517] RIP  [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.418886]  RSP <ffff889ff43cfd00>
    [2357701.420205] CR2: 00000000000000a8
    

Environment

  • Red Hat Enterprise Linux 7.3
  • QLogic QLE3262 Converged Network Adapter

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In