IBM System z server got crashed after 'QDIO problem occurred' errors

Solution Verified - Updated -

Issue

  • During the storage connectivity issues, couple of IBM System z servers got crashed with following errors:

    sd 1:0:0:120: [sday] FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=0s
    sd 1:0:0:120: [sday] CDB: Test Unit Ready 00 00 00 00 00 00
    zfcp 0.0.0b00: Setting up the QDIO connection to the FCP adapter failed
    qdio: 0.0.0b00 ZFCP on SC 3 using AI:1 QEBSM:1 PRI:1 TDD:1 SIGA: W A 
    [...]
    zfcp 0.0.0b00: A QDIO problem occurred
    zfcp 0.0.0b00: A QDIO problem occurred
    zfcp 0.0.0b00: A QDIO problem occurred
    zfcp 0.0.0b00: A QDIO problem occurred
    Unable to handle kernel pointer dereference at virtual kernel address 0024e3a001100000
    Oops: 0038 [#1] SMP 
    Modules linked in: softdog rpcsec_gss_krb5 auth_rpcgss nfsv4 [...]
    CPU: 4 PID: 1043 Comm: zfcperp0.0.0b00 Kdump: loaded Not tainted 3.10.0-1160.95.1.el7.s390x #1
    task: 0000001fce53a340 ti: 0000001fce304000 task.ti: 0000001fce304000
    Krnl PSW : 0704e00180000000 000003ff80607a24 (zfcp_fsf_req_complete+0x5cc/0x928 [zfcp])
               R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA:3
               Krnl GPRS: 000000000015e640 0024e3a001100004 0000000000000000 0000000000000200
               0000001fd0a54600 0000000000000000 0000001fd0a7e800 070000000017c59a
               0000000000000000 0000001fd0a7e800 0000000000000000 0000001fd4aef000
               0000001fd0a7e800 000003ff80610108 000003ff806074b4 0000001fce307b70
    Krnl Code: 000003ff80607a14: a7f4fe2e          brc     15,3ff80607670
               000003ff80607a18: e310a1b80004       lg      %r1,440(%r10)
              #000003ff80607a1e: e31010100004       lg      %r1,16(%r1)
              >000003ff80607a24: e55c10180002       chsi    24(%r1),2
               000003ff80607a2a: a7240079           brc     2,3ff80607b1c
               000003ff80607a2e: e340f0a00004       lg      %r4,160(%r15)
               000003ff80607a34: 58104050           l       %r1,80(%r4)
               000003ff80607a38: a7f4fd72           brc     15,3ff8060751c
    Call Trace:
    ([<000003ff806074b4>] zfcp_fsf_req_complete+0x5c/0x928 [zfcp])
     [<000003ff80608134>] zfcp_fsf_req_dismiss_all+0x124/0x168 [zfcp]
     [<000003ff80600118>] zfcp_erp_adapter_strategy_close.isra.2+0x50/0x90 [zfcp]
     [<000003ff80601254>] zfcp_erp_strategy_do_action+0x25c/0x7c8 [zfcp]
     [<000003ff80601e10>] zfcp_erp_strategy+0x268/0xae0 [zfcp]
     [<000003ff8060274e>] zfcp_erp_thread+0xc6/0x238 [zfcp]
     [<000000000017b652>] kthread+0xea/0xf8
     [<0000000000765d56>] kernel_thread_starter+0x6/0x10
     [<0000000000765d50>] kernel_thread_starter+0x0/0x10
    Last Breaking-Event-Address:
     [<000003ff806074ce>] zfcp_fsf_req_complete+0x76/0x928 [zfcp]
    Kernel panic - not syncing: Fatal exception: panic_on_oops
    

Environment

  • Red Hat Enterprise Linux 7
    • kernel version < 3.10.0-1160.114.2.el7
  • Red Hat Enterprise Linux 8
    • kernel version < 4.18.0-477.10.1.el8_8
  • IBM S/390 and IBM System z
  • LUNs connected through zfcp host adapter

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content