RHV node rebooted during storage outage.

Solution Verified - Updated -

Issue

  • We had a failure on one of the IBM SVC storage cluster nodes, this caused a failure of few paths and RHV node was fenced/rebooted:

    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:8 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:8 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:8 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:8 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:13 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    kernel: qla2xxx [0000:09:00.1]-801c:2: Abort command issued nexus=2:11:9 -- 2003.
    [...]
    sanlock[4377]: 2020-11-08 00:11:34 6098082 [4377]: s22 check_our_lease warning 63 last_success 6098019
    sanlock[4377]: 2020-11-08 00:11:34 6098082 [4377]: s20 check_our_lease warning 70 last_success 6098012
    sanlock[4377]: 2020-11-08 00:11:34 6098082 [4377]: s19 check_our_lease warning 74 last_success 6098008
    multipathd: checker failed path 65:224 in map 3600507681082008908000000000005f2
    multipathd: 3600507681082008908000000000005f2: remaining active paths: 1
    kernel: device-mapper: multipath: Failing path 65:224.
    kernel: device-mapper: multipath: Failing path 65:240.
    kernel: device-mapper: multipath: Failing path 66:128.
    [...]
    

Environment

  • Red Hat Enterprise Linux 7.7, 7.8
  • Red Hat Virtualization Host (RHV-H)
    • versions >=4.3.5 and <=4.3.10
  • kernel-3.10.0
    • versions >=3.10.0-1062.el7.x86_64 and <=3.10.0-1127.13.1.el7.x86_64
  • Fibre Channel SAN
    • QLogic FC HBAs

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content