System crashed due to list corruption in iscsi_queuecommand.

Solution Verified - Updated -

Issue

  • System encountered list corruption and crashed/rebooted.
[1028762.501040] list_add corruption. prev->next should be next (c00020354be673f8), but was c000203963d4f880. (prev=c000203963d4f880).
[1028762.501072] ------------[ cut here ]------------
[1028762.501081] WARNING: CPU: 21 PID: 28260 at lib/list_debug.c:28 __list_add_valid+0x94/0xc0
[1028762.501083] Kernel panic - not syncing: panic_on_warn set ...

[1028762.501147] CPU: 21 PID: 28260 Comm: kdmwork-253:11 Kdump: loaded Not tainted 4.14.0-115.13.1.el7a.ppc64le #1
[1028762.501265] Call Trace:
[1028762.501282] [c000203936db3370] [c000000000c217ac] dump_stack+0xb0/0xf4 (unreliable)
[1028762.501388] [c000203936db33b0] [c000000000138834] panic+0x150/0x33c
[1028762.501464] [c000203936db3440] [c0000000001386e4] panic+0x0/0x33c
[1028762.501547] [c000203936db34d0] [c000000000c204d4] report_bug+0x184/0x220
[1028762.501622] [c000203936db3550] [c00000000002a36c] program_check_exception+0x31c/0x420
[1028762.501702] [c000203936db35d0] [c000000000008e94] program_check_common+0x134/0x140
[1028762.501801] --- interrupt: 700 at __list_add_valid+0x94/0xc0
                     LR = __list_add_valid+0x90/0xc0
[1028762.501926] [c000203936db3920] [c00800000d689d80] iscsi_queuecommand+0x420/0x530 [libiscsi]
[1028762.502044] [c000203936db39e0] [c000000000856444] scsi_dispatch_cmd+0xe4/0x300
[1028762.502131] [c000203936db3a60] [c00000000085a488] scsi_request_fn+0x5e8/0x9e0
[1028762.502241] [c000203936db3b80] [c00000000061c00c] __blk_run_queue+0x6c/0xb0
[1028762.502338] [c000203936db3bb0] [c000000000617638] __elv_add_request+0x1b8/0x540
[1028762.502453] [c000203936db3c40] [c000000000623d4c] blk_insert_cloned_request+0x1cc/0x280
[1028762.502566] [c000203936db3cc0] [c008000011317904] map_tio_request+0x1c4/0x370 [dm_mod]
[1028762.502665] [c000203936db3d60] [c000000000172260] kthread_worker_fn+0xb0/0x290
[1028762.502765] [c000203936db3dc0] [c000000000171618] kthread+0x168/0x1b0
[1028762.502867] [c000203936db3e30] [c00000000000b65c] ret_from_kernel_thread+0x5c/0x80
[1028762.503017] Sending IPI to other CPUs
[1028764.111921] IPI complete
[1028766.209118] kexec: Starting switchover sequence.
  • In some cases, system can crash in libiscsi_tcp.c:
PANIC: "kernel BUG at drivers/scsi/libiscsi_tcp.c:979!"

PID: 2426   TASK: c00000079e618700  CPU: 7   COMMAND: "kworker/u32:0"
 #0 [c00000032d07f4a0] crash_kexec at c000000000234ce0
 #1 [c00000032d07f4e0] oops_end at c0000000000284d8
 #2 [c00000032d07f560] _exception at c0000000000299a4
 #3 [c00000032d07f6f0] program_check_exception at c000000000029e6c
 #4 [c00000032d07f770] program_check_common at c000000000008da4
 Program Check [700] exception frame:
 R0:  c00800000cda9690    R1:  c00000032d07fa60    R2:  c00800000ce1c000   
 R3:  c0000007ee418800    R4:  c00000032fd6d16a    R5:  000000000000000a   
 R6:  000000000000a047    R7:  0000000000000000    R8:  00000000001fd998   
 R9:  c0000007ee418898    R10: 0000000000000008    R11: c00800000cdab798   
 R12: c00800000ce10400    R13: c000000007a84d00    R14: c0000000001708a8   
 R15: c00000046decc540    R16: c0000007fd200380    R17: c00000000137c4b0   
 R18: c0000007fd200078    R19: c0000007fd200020    R20: 0000000000000000   
 R21: 0000000000000000    R22: fffffffffffffef7    R23: 0000000000000003   
 R24: c00000079e090408    R25: 0000000000000000    R26: 000000000000b000   
 R27: c00000032fd6d160    R28: c00000079e091420    R29: c00000079e090348   
 R30: c00000079e090348    R31: c0000007ee418800   
 NIP: c00800000ce1044c    MSR: 800000000282b033    OR3: c00800000ce10420
 CTR: c00800000ce10400    LR:  c00800000cda9690    XER: 0000000020040000
 CCR: 0000000028072e48    MQ:  0000000000000001    DAR: 0000000000000000
 DSISR: 0000000000000000     Syscall Result: 0000000000000000
 [NIP  : iscsi_tcp_task_init+76]
 [LR   : iscsi_prep_scsi_cmd_pdu+832]
 #5 [c00000032d07fa60] iscsi_tcp_task_init at c00800000ce1044c [libiscsi_tcp]
 #6 [c00000032d07fad0] iscsi_prep_scsi_cmd_pdu at c00800000cda9690 [libiscsi]  (unreliable)
 #7 [c00000032d07fbc0] iscsi_xmitworker at c00800000cdaa278 [libiscsi]
 #8 [c00000032d07fc80] process_one_work at c000000000163acc
 #9 [c00000032d07fd20] worker_thread at c000000000166aec

Environment

  • Red Hat Enterprise Linux 7.6
    • 4.14.0-115.13.1.el7a.ppc64le
  • iSCSI SAN

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content