System crash due to lpfc double-completion following I/O abort handling

Solution Unverified - Updated -

Issue

  • System crash due to lpfc double-completion following I/O abort handling:
lpfc 0000:b1:00.0: 155: [84102.235676] 0:0331 IOCB wake signaled
lpfc 0000:b1:00.0: 0:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 0 failed (3, 34603030) iocb_flag x6
lpfc 0000:b1:00.0: start 156 end 160 cnt 4
lpfc 0000:b1:00.0: 156: [84102.236091] 0:0373 FCP CQE cmpl: status=x3: CQE: 1a7e0300 00000000 02230016 90010000
lpfc 0000:b1:00.0: 157: [84102.236092] 0:(0):9034 FCP cmd x2a failed <1/0> status: x3 result: x16 sid: xc8200 did: x75901 oxid: x378 Data: x2230016 x0 x1a7e
lpfc 0000:b1:00.0: 158: [84102.236093] 0:(0):9039 Iodone <1/0> cmd xff13a78398ebde78, error xd0000 SNS x0 x0 LBA x34b6e7b8 Data: x1 x0
lpfc 0000:b1:00.0: 159: [84102.236121] 0:3017 ABORT_XRI_CN completing on rpi x0 original iotag x378, abort cmd iotag x37c status 0x0, reason 0x9
lpfc 0000:b1:00.0: 0:(0):0713 SCSI layer issued Device Reset (1, 0) return x2003
scsi_io_completion_action: 14 callbacks suppressed
sd 11:0:1:0: [sdf] tag#935 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=82s
sd 11:0:1:0: [sdf] tag#935 CDB: Write(10) 2a 00 31 c1 78 58 00 0a 00 00
print_req_error: 15 callbacks suppressed
blk_update_request: I/O error, dev sdf, sector 834762840 op 0x1:(WRITE) flags 0x104200 phys_seg 320 prio class 0
sd 11:0:1:0: rejecting I/O to offline device
sd 11:0:1:0: [sdf] tag#2318 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=97s
sd 11:0:1:0: [sdf] tag#2318 CDB: Write(10) 2a 00 31 cf 9f 08 00 0a 00 00
....
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
sd 11:0:1:0: [sdf] tag#4162 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
PGD 0 
sd 11:0:1:0: [sdf] tag#4162 CDB: Write(10) 2a 00 34 b7 07 b8 00 09 08 00
Oops: 0000 [#1] SMP NOPTI
blk_update_request: I/O error, dev sdf, sector 884410296 op 0x1:(WRITE) flags 0x104200 phys_seg 289 prio class 0
CPU: 21 PID: 0 Comm: swapper/21 Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-425.13.1.el8.rhel.12777.x86_64 #1
sd 11:0:1:0: [sdf] tag#4203 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
Hardware name: Dell Inc. PowerEdge R750/04V528, BIOS 1.8.2 09/14/2022
RIP: 0010:dma_direct_unmap_sg+0x46/0x1b0
sd 11:0:1:0: [sdf] tag#4203 CDB: Write(10) 2a 00 34 b6 e1 b8 00 06 00 00
Code: 18 4c 89 04 24 85 d2 0f 8e c0 00 00 00 49 89 fc 48 89 f0 89 d3 41 89 cd 45 31 f6 48 8b 35 aa 49 6c 01 49 8b 94 24 90 02 00 00 <44> 8b 50 18 48 8b 48 10 48 f7 d6 48 85 d2 0f 84 0d 01 00 00 48 8b
blk_update_request: I/O error, dev sdf, sector 884400568 op 0x1:(WRITE) flags 0x104200 phys_seg 192 prio class 0
RSP: 0018:ff1e6ebe193f8b10 EFLAGS: 00010246
sd 11:0:1:0: [sdf] tag#4207 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
sd 11:0:1:0: [sdf] tag#4207 CDB: Write(10) 2a 00 34 b6 e7 b8 00 0a 00 00
RAX: 0000000000000000 RBX: 0000000000000140 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff13a7828f76d0b8
blk_update_request: I/O error, dev sdf, sector 884402104 op 0x1:(WRITE) flags 0x104200 phys_seg 320 prio class 0
RBP: 0000000000000000 R08: 0000000000000020 R09: 000000000002a780
sd 11:0:1:0: [sdf] tag#4212 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
R10: 001a60df257be1b7 R11: ff13a782925adeff R12: ff13a7828f76d0b8
sd 11:0:1:0: [sdf] tag#4212 CDB: Write(10) 2a 00 33 0f 36 f8 00 06 00 00
R13: 0000000000000001 R14: 0000000000000000 R15: ff1e6ebe193f8e58
FS:  0000000000000000(0000) GS:ff13a8007f080000(0000) knlGS:0000000000000000
blk_update_request: I/O error, dev sdf, sector 856635128 op 0x1:(WRITE) flags 0x104200 phys_seg 192 prio class 0
....
Call Trace:
 <IRQ>
 lpfc_scsi_unprep_dma_buf+0x6d/0x70 [lpfc]
 lpfc_fcp_io_cmd_wqe_cmpl+0x1c8/0xf40 [lpfc]
 lpfc_sli4_fp_handle_fcp_wcqe.isra.27+0x12d/0x2d0 [lpfc]
 ? vsnprintf+0x105/0x520
 ? sched_clock+0x5/0x10
 ? sched_clock_cpu+0xc/0xb0
 ? lpfc_dbg_print+0xaf/0xe0 [lpfc]
 ? mempool_alloc+0x67/0x180
 ? lpfc_put_cmd_rsp_buf_per_hdwq+0xca/0xe0 [lpfc]
 ? lpfc_release_io_buf+0x19e/0x220 [lpfc]
 ? lpfc_sli4_io_xri_aborted+0x30b/0x4b0 [lpfc]
 ? lpfc_sli4_sp_handle_abort_xri_wcqe.isra.53+0x4e/0x190 [lpfc]
 ? lpfc_sli4_fp_handle_cqe+0x194/0x460 [lpfc]
 lpfc_sli4_fp_handle_cqe+0x194/0x460 [lpfc]
 __lpfc_sli4_process_cq+0x109/0x220 [lpfc]
 ? lpfc_sli4_fp_handle_fcp_wcqe.isra.27+0x2d0/0x2d0 [lpfc]
 __lpfc_sli4_hba_process_cq+0x3c/0x100 [lpfc]
 lpfc_cq_poll_hdler+0x16/0x20 [lpfc]
 irq_poll_softirq+0x76/0x110
 __do_softirq+0xdc/0x2cf
....

Environment

  • Red Hat Enterprise Linux 8

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content