RHEL VM on IBM LPAR gets crashed with IO errors

Solution Verified - Updated -

Issue

  • RHEL VM on IBM LPAR gets crashed with IO errors:

    device-mapper: multipath: Failing path 68:224.
    device-mapper: multipath: Failing path 69:0.
    sd 3:0:2:10: rejecting I/O to offline device
    device-mapper: multipath: Failing path 68:96.
    device-mapper: multipath: Failing path 66:144.
    device-mapper: multipath: Failing path 67:160.
    [...]
    sd 3:0:2:1: Successfully cancelled outstanding commands
    sd 3:0:2:4: Cancelling outstanding commands.
    sd 3:0:3:4: Successfully cancelled outstanding commands
    ibmvfc 30000008: Timed out waiting for aborted commands
    ibmvfc 30000008: Initiating host LIP. Resetting connection
    list_del corruption, c0000007d6365460->next is LIST_POISON1 (0000000000000100)
    ------------[ cut here ]------------
    WARNING: CPU: 0 PID: 21910 at lib/list_debug.c:53 list_del+0x90/0xf0
    [...]
    CPU: 0 PID: 21910 Comm: kworker/0:1 Kdump: loaded Not tainted 3.10.0-1062.12.1.el7.ppc64le #1
    Workqueue: fc_dl_3 fc_timeout_fail_rport_io [scsi_transport_fc]
    task: c0000006d4e0c570 ti: c00000006a998000 task.ti: c00000006a998000
    NIP: c0000000005c1a60 LR: c0000000005c1a5c CTR: c0000000005a2c70
    REGS: c00000006a99b720 TRAP: 0700   Not tainted  (3.10.0-1062.12.1.el7.ppc64le)
    MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 480f2e88  XER: 20000000
    CFAR: c000000000af2164 SOFTE: 0 
    GPR00: c0000000005c1a5c c00000006a99b9a0 c00000000148a200 000000000000004e 
    GPR04: 0000000000000000 000000000000000e 0000000030303030 0000000000000000 
    GPR08: 0000000000000001 0000000000000007 0000000000000006 53494f505f545349 
    GPR12: 0000000000002200 c000000007b30000 c000000000139d28 c000000062ddbb40 
    GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
    GPR20: 0000000000000001 c0000000013f4aaa 0000000000000001 ffffffffffffee00 
    GPR24: 0000000000070000 c0000000014c2280 ffffffffffffffff c0000007e6ecb7a8 
    GPR28: c0000007d755ab0c 0000000000000100 c0000007d6246740 c0000007d6365460 
    NIP [c0000000005c1a60] list_del+0x90/0xf0
    LR [c0000000005c1a5c] list_del+0x8c/0xf0
    Call Trace:
    [c00000006a99b9a0] [c0000000005c1a5c] list_del+0x8c/0xf0 (unreliable)
    [c00000006a99ba10] [d000000003fd6280] ibmvfc_purge_requests+0x100/0x250 [ibmvfc]
    [c00000006a99bac0] [d000000003fd6768] ibmvfc_issue_fc_host_lip+0x88/0x150 [ibmvfc]
    [c00000006a99bb40] [d000000003fd83f8] ibmvfc_terminate_rport_io+0x328/0x400 [ibmvfc]
    [c00000006a99bc10] [d000000003d986fc] fc_timeout_fail_rport_io+0xbc/0x100 [scsi_transport_fc]
    [c00000006a99bc40] [c00000000012ce9c] process_one_work+0x1dc/0x680
    [c00000006a99bce0] [c00000000012d4e0] worker_thread+0x1a0/0x520
    [c00000006a99bd80] [c000000000139e14] kthread+0xf4/0x100
    [c00000006a99be30] [c00000000000a62c] ret_from_kernel_thread+0x5c/0x70
    

Environment

  • Red Hat Enterprise Linux 7.7
  • VM created on IBM LPAR
  • IBM 2145 storage LUNs connected through Virtual FC adapters

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In