System crashed after dm-multipath path failures

Solution In Progress - Updated -

Issue

  • The FCoE HBA present on system had encountered a fw hang after which paths to multipath devices were failed. Shortly after the path failures, system got rebooted with following panic messages:

    [2321378.786537] qla2xxx [0000:81:00.7]-6006:4: Detect abort  needed.
    [2321378.786539] qla2xxx [0000:81:00.7]-6007:4: Firmware hung.
    [2321378.787040] qla2xxx [0000:81:00.7]-b02f:4: HW State: NEED RESET
    [2321378.787045] qla2xxx [0000:81:00.7]-009b:4: Device state is 0x4 = Need Reset.
    [2321378.787047] qla2xxx [0000:81:00.7]-009d:4: Device state is 0x4 = Need Reset.
    [2321378.787050] qla2xxx [0000:81:00.7]-00af:4: Performing ISP error recovery - ha=ffff88befa5b4000.
    [2321378.787153] qla2xxx [0000:81:00.7]-00b4:4: Done chip reset cleanup.
    [2321378.794558] qlcnic 0000:81:00.0: Pause control frames disabled on all ports
    [2321378.794569] qlcnic 0000:81:00.0: firmware hang detected
    [2321378.794589] qlcnic 0000:81:00.1: Pause control frames disabled on all ports
    [2321378.794594] qlcnic 0000:81:00.1: firmware hang detected
    [2321378.794603] qlcnic 0000:81:00.1: Dumping hw/fw registers
    ...
    [2357691.366302] qla2xxx [0000:0b:00.7]-8040:2: PCI/Register disconnect, exiting
    ...
    [2357691.366303] qla2xxx [0000:0b:00.6]-015b:1: Disabling adapter.
    [2357691.366304] qla2xxx [0000:0b:00.7]-8041:2: PCI/Register disconnect, exiting
    ...
    [2357691.366308] qla2xxx [0000:0b:00.7]-015b:2: Disabling adapter.
    ...
    [2357701.370522] device-mapper: multipath: Failing path 70:208.
    [2357701.370523] device-mapper: multipath: Failing path 71:32.
    [2357701.373447] IP: [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.373814] PGD 0 
    [2357701.374164] Oops: 0002 [#1] SMP 
    [2357701.377000] sd 2:0:3:12: alua: Detached
    [...]
    [2357701.380637] CPU: 7 PID: 4300 Comm: systemd-udevd Tainted: G           OE  ------------   3.10.0-514.26.1.el7.x86_64 #1
    [2357701.381401] Hardware name: LENOVO System x3650 M5: -[8871AC1]-/01GR174, BIOS -[TCE130J-2.40]- 04/11/2017
    [2357701.382158] task: ffff88bee0954e70 ti: ffff889ff43cc000 task.ti: ffff889ff43cc000
    [2357701.382918] RIP: 0010:[<ffffffff8168eb1f>]  [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.383721] RSP: 0018:ffff889ff43cfd00  EFLAGS: 00010046
    [...]
    [2357701.393097] Stack:
    [2357701.393955]  ffff889ff43cfd38 ffffffff810b1757 ffff885ef6289000 0000000000000000
    [2357701.394877]  ffff885ef6289000 ffff88bef65aaa40 ffff885ef6289000 ffff889ff43cfd88
    [2357701.395823]  ffffffff81455d0d 0000000000000000 ffff88bee0954e70 ffffffff810b1b20
    [2357701.396741] Call Trace:
    [2357701.397644]  [<ffffffff810b1757>] prepare_to_wait+0x27/0x90
    [2357701.398527]  [<ffffffff81455d0d>] scsi_block_when_processing_errors+0xdd/0x140
    [2357701.399403]  [<ffffffff810b1b20>] ? wake_up_atomic_t+0x30/0x30
    [2357701.400288]  [<ffffffff8145398c>] scsi_nonblockable_ioctl+0xcc/0xf0
    [2357701.401472]  [<ffffffffa012697e>] sd_ioctl+0x6e/0x140 [sd_mod]
    [2357701.402782]  [<ffffffff812fd1c2>] __blkdev_driver_ioctl+0x22/0x30
    [2357701.404075]  [<ffffffffa0068042>] dm_blk_ioctl+0x82/0x90 [dm_mod]
    [2357701.405362]  [<ffffffff812fdb40>] blkdev_ioctl+0x270/0x980
    [2357701.406663]  [<ffffffff8121f15e>] ? mntput_no_expire+0x3e/0x120
    [2357701.407986]  [<ffffffff8123a7b1>] block_ioctl+0x41/0x50
    [2357701.409310]  [<ffffffff81212855>] do_vfs_ioctl+0x2d5/0x4b0
    [2357701.410676]  [<ffffffff81200b3e>] ? ____fput+0xe/0x10
    [2357701.411992]  [<ffffffff81212ad1>] SyS_ioctl+0xa1/0xc0
    [2357701.413316]  [<ffffffff81697809>] system_call_fastpath+0x16/0x1b
    [2357701.414686] Code: df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 0f 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 <f0> 0f c1 17 89 d1 c1 e9 10 66 39 d1 75 05 4c 89 c0 5d c3 83 e1 
    [2357701.417517] RIP  [<ffffffff8168eb1f>] _raw_spin_lock_irqsave+0x1f/0x60
    [2357701.418886]  RSP <ffff889ff43cfd00>
    [2357701.420205] CR2: 00000000000000a8
    

Environment

  • Red Hat Enterprise Linux 7.3
  • QLogic QLE3262 Converged Network Adapter

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content