System crash due to lpfc double-completion following I/O abort handling
Issue
- System crash due to
lpfcdouble-completion following I/O abort handling:
lpfc 0000:b1:00.0: 155: [84102.235676] 0:0331 IOCB wake signaled
lpfc 0000:b1:00.0: 0:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 0 failed (3, 34603030) iocb_flag x6
lpfc 0000:b1:00.0: start 156 end 160 cnt 4
lpfc 0000:b1:00.0: 156: [84102.236091] 0:0373 FCP CQE cmpl: status=x3: CQE: 1a7e0300 00000000 02230016 90010000
lpfc 0000:b1:00.0: 157: [84102.236092] 0:(0):9034 FCP cmd x2a failed <1/0> status: x3 result: x16 sid: xc8200 did: x75901 oxid: x378 Data: x2230016 x0 x1a7e
lpfc 0000:b1:00.0: 158: [84102.236093] 0:(0):9039 Iodone <1/0> cmd xff13a78398ebde78, error xd0000 SNS x0 x0 LBA x34b6e7b8 Data: x1 x0
lpfc 0000:b1:00.0: 159: [84102.236121] 0:3017 ABORT_XRI_CN completing on rpi x0 original iotag x378, abort cmd iotag x37c status 0x0, reason 0x9
lpfc 0000:b1:00.0: 0:(0):0713 SCSI layer issued Device Reset (1, 0) return x2003
scsi_io_completion_action: 14 callbacks suppressed
sd 11:0:1:0: [sdf] tag#935 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=82s
sd 11:0:1:0: [sdf] tag#935 CDB: Write(10) 2a 00 31 c1 78 58 00 0a 00 00
print_req_error: 15 callbacks suppressed
blk_update_request: I/O error, dev sdf, sector 834762840 op 0x1:(WRITE) flags 0x104200 phys_seg 320 prio class 0
sd 11:0:1:0: rejecting I/O to offline device
sd 11:0:1:0: [sdf] tag#2318 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=97s
sd 11:0:1:0: [sdf] tag#2318 CDB: Write(10) 2a 00 31 cf 9f 08 00 0a 00 00
....
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
sd 11:0:1:0: [sdf] tag#4162 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
PGD 0
sd 11:0:1:0: [sdf] tag#4162 CDB: Write(10) 2a 00 34 b7 07 b8 00 09 08 00
Oops: 0000 [#1] SMP NOPTI
blk_update_request: I/O error, dev sdf, sector 884410296 op 0x1:(WRITE) flags 0x104200 phys_seg 289 prio class 0
CPU: 21 PID: 0 Comm: swapper/21 Kdump: loaded Tainted: P OE --------- - - 4.18.0-425.13.1.el8.rhel.12777.x86_64 #1
sd 11:0:1:0: [sdf] tag#4203 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
Hardware name: Dell Inc. PowerEdge R750/04V528, BIOS 1.8.2 09/14/2022
RIP: 0010:dma_direct_unmap_sg+0x46/0x1b0
sd 11:0:1:0: [sdf] tag#4203 CDB: Write(10) 2a 00 34 b6 e1 b8 00 06 00 00
Code: 18 4c 89 04 24 85 d2 0f 8e c0 00 00 00 49 89 fc 48 89 f0 89 d3 41 89 cd 45 31 f6 48 8b 35 aa 49 6c 01 49 8b 94 24 90 02 00 00 <44> 8b 50 18 48 8b 48 10 48 f7 d6 48 85 d2 0f 84 0d 01 00 00 48 8b
blk_update_request: I/O error, dev sdf, sector 884400568 op 0x1:(WRITE) flags 0x104200 phys_seg 192 prio class 0
RSP: 0018:ff1e6ebe193f8b10 EFLAGS: 00010246
sd 11:0:1:0: [sdf] tag#4207 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
sd 11:0:1:0: [sdf] tag#4207 CDB: Write(10) 2a 00 34 b6 e7 b8 00 0a 00 00
RAX: 0000000000000000 RBX: 0000000000000140 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff13a7828f76d0b8
blk_update_request: I/O error, dev sdf, sector 884402104 op 0x1:(WRITE) flags 0x104200 phys_seg 320 prio class 0
RBP: 0000000000000000 R08: 0000000000000020 R09: 000000000002a780
sd 11:0:1:0: [sdf] tag#4212 FAILED Result: hostbyte=DID_REQUEUE driverbyte=DRIVER_OK cmd_age=92s
R10: 001a60df257be1b7 R11: ff13a782925adeff R12: ff13a7828f76d0b8
sd 11:0:1:0: [sdf] tag#4212 CDB: Write(10) 2a 00 33 0f 36 f8 00 06 00 00
R13: 0000000000000001 R14: 0000000000000000 R15: ff1e6ebe193f8e58
FS: 0000000000000000(0000) GS:ff13a8007f080000(0000) knlGS:0000000000000000
blk_update_request: I/O error, dev sdf, sector 856635128 op 0x1:(WRITE) flags 0x104200 phys_seg 192 prio class 0
....
Call Trace:
<IRQ>
lpfc_scsi_unprep_dma_buf+0x6d/0x70 [lpfc]
lpfc_fcp_io_cmd_wqe_cmpl+0x1c8/0xf40 [lpfc]
lpfc_sli4_fp_handle_fcp_wcqe.isra.27+0x12d/0x2d0 [lpfc]
? vsnprintf+0x105/0x520
? sched_clock+0x5/0x10
? sched_clock_cpu+0xc/0xb0
? lpfc_dbg_print+0xaf/0xe0 [lpfc]
? mempool_alloc+0x67/0x180
? lpfc_put_cmd_rsp_buf_per_hdwq+0xca/0xe0 [lpfc]
? lpfc_release_io_buf+0x19e/0x220 [lpfc]
? lpfc_sli4_io_xri_aborted+0x30b/0x4b0 [lpfc]
? lpfc_sli4_sp_handle_abort_xri_wcqe.isra.53+0x4e/0x190 [lpfc]
? lpfc_sli4_fp_handle_cqe+0x194/0x460 [lpfc]
lpfc_sli4_fp_handle_cqe+0x194/0x460 [lpfc]
__lpfc_sli4_process_cq+0x109/0x220 [lpfc]
? lpfc_sli4_fp_handle_fcp_wcqe.isra.27+0x2d0/0x2d0 [lpfc]
__lpfc_sli4_hba_process_cq+0x3c/0x100 [lpfc]
lpfc_cq_poll_hdler+0x16/0x20 [lpfc]
irq_poll_softirq+0x76/0x110
__do_softirq+0xdc/0x2cf
....
Environment
- Red Hat Enterprise Linux 8
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.