RHEL6: Kernel panic in tcp_xmit_retransmit_queue() or tcp_mark_head_lost()

Solution Verified - Updated -

Issue

Server panics in tcp_xmit_retransmit_queue+128 with a trace like the following:

        [exception RIP: tcp_xmit_retransmit_queue+128]
        RIP: ffffffff8148c920  RSP: ffff8800282638c0  RFLAGS: 00010202
        RAX: 0000000000000086  RBX: ffff8801041876c0  RCX: 0000000000000003
        RDX: 000000000000002e  RSI: 00000000f1a4b83d  RDI: 0000000000000001
        RBP: ffff880028263910   R8: 0000000000000008   R9: 00000000f1a4b83d
        R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000
        R13: 0000000000000000  R14: ffff880104187788  R15: 0000000000000000
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
     #9 [ffff880028263918] tcp_fastretrans_alert at ffffffff81485f37
    #10 [ffff880028263968] tcp_ack at ffffffff81486d93
    #11 [ffff880028263a38] tcp_rcv_established at ffffffff81487d5d
    #12 [ffff880028263a98] tcp_v4_do_rcv at ffffffff8148fd53
    #13 [ffff880028263b38] tcp_v4_rcv at ffffffff814915ce
    #14 [ffff880028263bb8] ip_local_deliver_finish at ffffffff8146f2ed

Or like this one:

        [exception RIP: tcp_xmit_retransmit_queue+128]
        RIP: ffffffff8147fca0  RSP: ffff880028203850  RFLAGS: 00010246
        RAX: 0000000000000000  RBX: ffff880c0829b000  RCX: 0000000000000002
        RDX: 0000000000000000  RSI: ffff8805f89a4338  RDI: 0000000000000002
        RBP: ffff8800282038a0   R8: 000000000000000d   R9: 000000008b667a73
        R10: 0000000000000002  R11: 000000008b6be2e9  R12: 0000000000000000
        R13: 0000000000000000  R14: ffff880c0829b0c8  R15: ffff8805f89a4300
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
     #9 [ffff8800282038a8] tcp_ack at ffffffff81479e03
    #10 [ffff880028203978] tcp_rcv_established at ffffffff8147b138
    #11 [ffff8800282039d8] tcp_v4_do_rcv at ffffffff81483163
    #12 [ffff880028203a78] tcp_v4_rcv at ffffffff81484951
    #13 [ffff880028203af8] ip_local_deliver_finish at ffffffff814626bd
    #14 [ffff880028203b28] ip_local_deliver at ffffffff81462948
    #15 [ffff880028203b58] ip_rcv_finish at ffffffff81461e0d
    #16 [ffff880028203b98] ip_rcv at ffffffff81462395
    #17 [ffff880028203bd8] __netif_receive_skb at ffffffff8142c34b
    #18 [ffff880028203c38] netif_receive_skb at ffffffff8142e408
    #19 [ffff880028203c78] napi_skb_finish at ffffffff8142e510
    #20 [ffff880028203c98] napi_gro_receive at ffffffff81430b99
    #21 [ffff880028203cb8] bnx2x_rx_int at ffffffffa02e25d6 [bnx2x]
    #22 [ffff880028203e18] bnx2x_poll at ffffffffa02e346c [bnx2x]
    #23 [ffff880028203e68] net_rx_action at ffffffff81430cb3
    #24 [ffff880028203ec8] __do_softirq at ffffffff81072191
    #25 [ffff880028203f38] call_softirq at ffffffff8100c24c
    #26 [ffff880028203f50] do_softirq at ffffffff8100de85
    #27 [ffff880028203f70] irq_exit at ffffffff81071f75
    #28 [ffff880028203f80] do_IRQ at ffffffff814f

Or server panics in tcp_mark_head_lost+185 with a trace like the following:

PID: 0      TASK: ffffffff81a8d020  CPU: 0   COMMAND: "swapper"
 #0 [ffff8800368034f0] machine_kexec at ffffffff810321cb
 #1 [ffff880036803550] crash_kexec at ffffffff810b8f22
 #2 [ffff880036803620] oops_end at ffffffff814f0560
 #3 [ffff880036803650] no_context at ffffffff8104234b
 #4 [ffff8800368036a0] __bad_area_nosemaphore at ffffffff810425d5
 #5 [ffff8800368036f0] bad_area_nosemaphore at ffffffff810426a3
 #6 [ffff880036803700] __do_page_fault at ffffffff81042d5d
 #7 [ffff880036803820] do_page_fault at ffffffff814f253e
 #8 [ffff880036803850] page_fault at ffffffff814ef8f5
    [exception RIP: tcp_mark_head_lost+185]
    RIP: ffffffff81473ad9  RSP: ffff880036803900  RFLAGS: 00010202
    RAX: 0000000000000001  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff8803ccd01c00  RDI: ffff880823f927c0
    RBP: ffff880036803930   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000002  R11: 000000003fae2a5d  R12: ffff880823f927c0
    R13: ffff88038d1d32f8  R14: 0000000000000001  R15: ffff880823f92888
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff880036803938] tcp_ack at ffffffff814798dd
#10 [ffff880036803a08] tcp_rcv_established at ffffffff8147acf8
#11 [ffff880036803a68] tcp_v4_do_rcv at ffffffff81482d23
#12 [ffff880036803b08] tcp_v4_rcv at ffffffff81484511
#13 [ffff880036803b88] ip_local_deliver_finish at ffffffff8146229d
#14 [ffff880036803bb8] ip_local_deliver at ffffffff81462528
#15 [ffff880036803be8] ip_rcv_finish at ffffffff814619ed
#16 [ffff880036803c28] ip_rcv at ffffffff81461f75
#17 [ffff880036803c68] __netif_receive_skb at ffffffff8142bf7b
#18 [ffff880036803cc8] netif_receive_skb at ffffffff8142e028
#19 [ffff880036803d08] napi_skb_finish at ffffffff8142e130
#20 [ffff880036803d28] napi_gro_receive at ffffffff814307b9
#21 [ffff880036803d48] igb_poll at ffffffffa0153d07 [igb]
#22 [ffff880036803e68] net_rx_action at ffffffff814308d3
#23 [ffff880036803ec8] __do_softirq at ffffffff81072001
#24 [ffff880036803f38] call_softirq at ffffffff8100c24c
#25 [ffff880036803f50] do_softirq at ffffffff8100de85
#26 [ffff880036803f70] irq_exit at ffffffff81071de5
#27 [ffff880036803f80] do_IRQ at ffffffff814f4dc5

Environment

  • Red Hat Enterprise Linux (RHEL) 6
    • 6.1: earlier than kernel-2.6.32-131.32.1.el6
    • 6.2: earlier than kernel-2.6.32-220.26.1.el6
    • 6.3: earlier than kernel-2.6.32-279.8.1.el6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content