System crashed due to list corruption in iscsi_queuecommand.
Issue
- System encountered list corruption and crashed/rebooted.
[1028762.501040] list_add corruption. prev->next should be next (c00020354be673f8), but was c000203963d4f880. (prev=c000203963d4f880).
[1028762.501072] ------------[ cut here ]------------
[1028762.501081] WARNING: CPU: 21 PID: 28260 at lib/list_debug.c:28 __list_add_valid+0x94/0xc0
[1028762.501083] Kernel panic - not syncing: panic_on_warn set ...
[1028762.501147] CPU: 21 PID: 28260 Comm: kdmwork-253:11 Kdump: loaded Not tainted 4.14.0-115.13.1.el7a.ppc64le #1
[1028762.501265] Call Trace:
[1028762.501282] [c000203936db3370] [c000000000c217ac] dump_stack+0xb0/0xf4 (unreliable)
[1028762.501388] [c000203936db33b0] [c000000000138834] panic+0x150/0x33c
[1028762.501464] [c000203936db3440] [c0000000001386e4] panic+0x0/0x33c
[1028762.501547] [c000203936db34d0] [c000000000c204d4] report_bug+0x184/0x220
[1028762.501622] [c000203936db3550] [c00000000002a36c] program_check_exception+0x31c/0x420
[1028762.501702] [c000203936db35d0] [c000000000008e94] program_check_common+0x134/0x140
[1028762.501801] --- interrupt: 700 at __list_add_valid+0x94/0xc0
LR = __list_add_valid+0x90/0xc0
[1028762.501926] [c000203936db3920] [c00800000d689d80] iscsi_queuecommand+0x420/0x530 [libiscsi]
[1028762.502044] [c000203936db39e0] [c000000000856444] scsi_dispatch_cmd+0xe4/0x300
[1028762.502131] [c000203936db3a60] [c00000000085a488] scsi_request_fn+0x5e8/0x9e0
[1028762.502241] [c000203936db3b80] [c00000000061c00c] __blk_run_queue+0x6c/0xb0
[1028762.502338] [c000203936db3bb0] [c000000000617638] __elv_add_request+0x1b8/0x540
[1028762.502453] [c000203936db3c40] [c000000000623d4c] blk_insert_cloned_request+0x1cc/0x280
[1028762.502566] [c000203936db3cc0] [c008000011317904] map_tio_request+0x1c4/0x370 [dm_mod]
[1028762.502665] [c000203936db3d60] [c000000000172260] kthread_worker_fn+0xb0/0x290
[1028762.502765] [c000203936db3dc0] [c000000000171618] kthread+0x168/0x1b0
[1028762.502867] [c000203936db3e30] [c00000000000b65c] ret_from_kernel_thread+0x5c/0x80
[1028762.503017] Sending IPI to other CPUs
[1028764.111921] IPI complete
[1028766.209118] kexec: Starting switchover sequence.
- In some cases, system can crash in libiscsi_tcp.c:
PANIC: "kernel BUG at drivers/scsi/libiscsi_tcp.c:979!"
PID: 2426 TASK: c00000079e618700 CPU: 7 COMMAND: "kworker/u32:0"
#0 [c00000032d07f4a0] crash_kexec at c000000000234ce0
#1 [c00000032d07f4e0] oops_end at c0000000000284d8
#2 [c00000032d07f560] _exception at c0000000000299a4
#3 [c00000032d07f6f0] program_check_exception at c000000000029e6c
#4 [c00000032d07f770] program_check_common at c000000000008da4
Program Check [700] exception frame:
R0: c00800000cda9690 R1: c00000032d07fa60 R2: c00800000ce1c000
R3: c0000007ee418800 R4: c00000032fd6d16a R5: 000000000000000a
R6: 000000000000a047 R7: 0000000000000000 R8: 00000000001fd998
R9: c0000007ee418898 R10: 0000000000000008 R11: c00800000cdab798
R12: c00800000ce10400 R13: c000000007a84d00 R14: c0000000001708a8
R15: c00000046decc540 R16: c0000007fd200380 R17: c00000000137c4b0
R18: c0000007fd200078 R19: c0000007fd200020 R20: 0000000000000000
R21: 0000000000000000 R22: fffffffffffffef7 R23: 0000000000000003
R24: c00000079e090408 R25: 0000000000000000 R26: 000000000000b000
R27: c00000032fd6d160 R28: c00000079e091420 R29: c00000079e090348
R30: c00000079e090348 R31: c0000007ee418800
NIP: c00800000ce1044c MSR: 800000000282b033 OR3: c00800000ce10420
CTR: c00800000ce10400 LR: c00800000cda9690 XER: 0000000020040000
CCR: 0000000028072e48 MQ: 0000000000000001 DAR: 0000000000000000
DSISR: 0000000000000000 Syscall Result: 0000000000000000
[NIP : iscsi_tcp_task_init+76]
[LR : iscsi_prep_scsi_cmd_pdu+832]
#5 [c00000032d07fa60] iscsi_tcp_task_init at c00800000ce1044c [libiscsi_tcp]
#6 [c00000032d07fad0] iscsi_prep_scsi_cmd_pdu at c00800000cda9690 [libiscsi] (unreliable)
#7 [c00000032d07fbc0] iscsi_xmitworker at c00800000cdaa278 [libiscsi]
#8 [c00000032d07fc80] process_one_work at c000000000163acc
#9 [c00000032d07fd20] worker_thread at c000000000166aec
Environment
- Red Hat Enterprise Linux 7.6
- 4.14.0-115.13.1.el7a.ppc64le
- iSCSI SAN
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.