IBM System z server got crashed after 'QDIO problem occurred' errors
Issue
-
During the storage connectivity issues, couple of IBM System z servers got crashed with following errors:
sd 1:0:0:120: [sday] FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=0s sd 1:0:0:120: [sday] CDB: Test Unit Ready 00 00 00 00 00 00 zfcp 0.0.0b00: Setting up the QDIO connection to the FCP adapter failed qdio: 0.0.0b00 ZFCP on SC 3 using AI:1 QEBSM:1 PRI:1 TDD:1 SIGA: W A [...] zfcp 0.0.0b00: A QDIO problem occurred zfcp 0.0.0b00: A QDIO problem occurred zfcp 0.0.0b00: A QDIO problem occurred zfcp 0.0.0b00: A QDIO problem occurred Unable to handle kernel pointer dereference at virtual kernel address 0024e3a001100000 Oops: 0038 [#1] SMP Modules linked in: softdog rpcsec_gss_krb5 auth_rpcgss nfsv4 [...] CPU: 4 PID: 1043 Comm: zfcperp0.0.0b00 Kdump: loaded Not tainted 3.10.0-1160.95.1.el7.s390x #1 task: 0000001fce53a340 ti: 0000001fce304000 task.ti: 0000001fce304000 Krnl PSW : 0704e00180000000 000003ff80607a24 (zfcp_fsf_req_complete+0x5cc/0x928 [zfcp]) R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA:3 Krnl GPRS: 000000000015e640 0024e3a001100004 0000000000000000 0000000000000200 0000001fd0a54600 0000000000000000 0000001fd0a7e800 070000000017c59a 0000000000000000 0000001fd0a7e800 0000000000000000 0000001fd4aef000 0000001fd0a7e800 000003ff80610108 000003ff806074b4 0000001fce307b70 Krnl Code: 000003ff80607a14: a7f4fe2e brc 15,3ff80607670 000003ff80607a18: e310a1b80004 lg %r1,440(%r10) #000003ff80607a1e: e31010100004 lg %r1,16(%r1) >000003ff80607a24: e55c10180002 chsi 24(%r1),2 000003ff80607a2a: a7240079 brc 2,3ff80607b1c 000003ff80607a2e: e340f0a00004 lg %r4,160(%r15) 000003ff80607a34: 58104050 l %r1,80(%r4) 000003ff80607a38: a7f4fd72 brc 15,3ff8060751c Call Trace: ([<000003ff806074b4>] zfcp_fsf_req_complete+0x5c/0x928 [zfcp]) [<000003ff80608134>] zfcp_fsf_req_dismiss_all+0x124/0x168 [zfcp] [<000003ff80600118>] zfcp_erp_adapter_strategy_close.isra.2+0x50/0x90 [zfcp] [<000003ff80601254>] zfcp_erp_strategy_do_action+0x25c/0x7c8 [zfcp] [<000003ff80601e10>] zfcp_erp_strategy+0x268/0xae0 [zfcp] [<000003ff8060274e>] zfcp_erp_thread+0xc6/0x238 [zfcp] [<000000000017b652>] kthread+0xea/0xf8 [<0000000000765d56>] kernel_thread_starter+0x6/0x10 [<0000000000765d50>] kernel_thread_starter+0x0/0x10 Last Breaking-Event-Address: [<000003ff806074ce>] zfcp_fsf_req_complete+0x76/0x928 [zfcp] Kernel panic - not syncing: Fatal exception: panic_on_oops
Environment
- Red Hat Enterprise Linux 7
- kernel version <
3.10.0-1160.114.2.el7
- kernel version <
- Red Hat Enterprise Linux 8
- kernel version <
4.18.0-477.10.1.el8_8
- kernel version <
- IBM S/390 and IBM System z
- LUNs connected through zfcp host adapter
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.