RHEL VM on IBM LPAR gets crashed with IO errors

Solution Verified - Updated -

Issue

  • RHEL VM on IBM LPAR gets crashed with IO errors:

    device-mapper: multipath: Failing path 68:224.
    device-mapper: multipath: Failing path 69:0.
    sd 3:0:2:10: rejecting I/O to offline device
    device-mapper: multipath: Failing path 68:96.
    device-mapper: multipath: Failing path 66:144.
    device-mapper: multipath: Failing path 67:160.
    [...]
    sd 3:0:2:1: Successfully cancelled outstanding commands
    sd 3:0:2:4: Cancelling outstanding commands.
    sd 3:0:3:4: Successfully cancelled outstanding commands
    ibmvfc 30000008: Timed out waiting for aborted commands
    ibmvfc 30000008: Initiating host LIP. Resetting connection
    list_del corruption, c0000007d6365460->next is LIST_POISON1 (0000000000000100)
    ------------[ cut here ]------------
    WARNING: CPU: 0 PID: 21910 at lib/list_debug.c:53 list_del+0x90/0xf0
    [...]
    CPU: 0 PID: 21910 Comm: kworker/0:1 Kdump: loaded Not tainted 3.10.0-1062.12.1.el7.ppc64le #1
    Workqueue: fc_dl_3 fc_timeout_fail_rport_io [scsi_transport_fc]
    task: c0000006d4e0c570 ti: c00000006a998000 task.ti: c00000006a998000
    NIP: c0000000005c1a60 LR: c0000000005c1a5c CTR: c0000000005a2c70
    REGS: c00000006a99b720 TRAP: 0700   Not tainted  (3.10.0-1062.12.1.el7.ppc64le)
    MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 480f2e88  XER: 20000000
    CFAR: c000000000af2164 SOFTE: 0 
    GPR00: c0000000005c1a5c c00000006a99b9a0 c00000000148a200 000000000000004e 
    GPR04: 0000000000000000 000000000000000e 0000000030303030 0000000000000000 
    GPR08: 0000000000000001 0000000000000007 0000000000000006 53494f505f545349 
    GPR12: 0000000000002200 c000000007b30000 c000000000139d28 c000000062ddbb40 
    GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
    GPR20: 0000000000000001 c0000000013f4aaa 0000000000000001 ffffffffffffee00 
    GPR24: 0000000000070000 c0000000014c2280 ffffffffffffffff c0000007e6ecb7a8 
    GPR28: c0000007d755ab0c 0000000000000100 c0000007d6246740 c0000007d6365460 
    NIP [c0000000005c1a60] list_del+0x90/0xf0
    LR [c0000000005c1a5c] list_del+0x8c/0xf0
    Call Trace:
    [c00000006a99b9a0] [c0000000005c1a5c] list_del+0x8c/0xf0 (unreliable)
    [c00000006a99ba10] [d000000003fd6280] ibmvfc_purge_requests+0x100/0x250 [ibmvfc]
    [c00000006a99bac0] [d000000003fd6768] ibmvfc_issue_fc_host_lip+0x88/0x150 [ibmvfc]
    [c00000006a99bb40] [d000000003fd83f8] ibmvfc_terminate_rport_io+0x328/0x400 [ibmvfc]
    [c00000006a99bc10] [d000000003d986fc] fc_timeout_fail_rport_io+0xbc/0x100 [scsi_transport_fc]
    [c00000006a99bc40] [c00000000012ce9c] process_one_work+0x1dc/0x680
    [c00000006a99bce0] [c00000000012d4e0] worker_thread+0x1a0/0x520
    [c00000006a99bd80] [c000000000139e14] kthread+0xf4/0x100
    [c00000006a99be30] [c00000000000a62c] ret_from_kernel_thread+0x5c/0x70
    

Environment

  • Red Hat Enterprise Linux 7.7
  • VM created on IBM LPAR
  • IBM 2145 storage LUNs connected through Virtual FC adapters

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content