RHEL6: Kernel panic in tcp_xmit_retransmit_queue() or tcp_mark_head_lost

Solution Verified - Updated -

Issue

  • Server panics in tcp_xmit_retransmit_queue+128 with a trace like the following:
        [exception RIP: tcp_xmit_retransmit_queue+128]
        RIP: ffffffff8148c920  RSP: ffff8800282638c0  RFLAGS: 00010202
        RAX: 0000000000000086  RBX: ffff8801041876c0  RCX: 0000000000000003
        RDX: 000000000000002e  RSI: 00000000f1a4b83d  RDI: 0000000000000001
        RBP: ffff880028263910   R8: 0000000000000008   R9: 00000000f1a4b83d
        R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000
        R13: 0000000000000000  R14: ffff880104187788  R15: 0000000000000000
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
     #9 [ffff880028263918] tcp_fastretrans_alert at ffffffff81485f37
    #10 [ffff880028263968] tcp_ack at ffffffff81486d93
    #11 [ffff880028263a38] tcp_rcv_established at ffffffff81487d5d
    #12 [ffff880028263a98] tcp_v4_do_rcv at ffffffff8148fd53
    #13 [ffff880028263b38] tcp_v4_rcv at ffffffff814915ce
    #14 [ffff880028263bb8] ip_local_deliver_finish at ffffffff8146f2ed
  • Or like this one:
        [exception RIP: tcp_xmit_retransmit_queue+128]
        RIP: ffffffff8147fca0  RSP: ffff880028203850  RFLAGS: 00010246
        RAX: 0000000000000000  RBX: ffff880c0829b000  RCX: 0000000000000002
        RDX: 0000000000000000  RSI: ffff8805f89a4338  RDI: 0000000000000002
        RBP: ffff8800282038a0   R8: 000000000000000d   R9: 000000008b667a73
        R10: 0000000000000002  R11: 000000008b6be2e9  R12: 0000000000000000
        R13: 0000000000000000  R14: ffff880c0829b0c8  R15: ffff8805f89a4300
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
     #9 [ffff8800282038a8] tcp_ack at ffffffff81479e03
    #10 [ffff880028203978] tcp_rcv_established at ffffffff8147b138
    #11 [ffff8800282039d8] tcp_v4_do_rcv at ffffffff81483163
    #12 [ffff880028203a78] tcp_v4_rcv at ffffffff81484951
    #13 [ffff880028203af8] ip_local_deliver_finish at ffffffff814626bd
    #14 [ffff880028203b28] ip_local_deliver at ffffffff81462948
    #15 [ffff880028203b58] ip_rcv_finish at ffffffff81461e0d
    #16 [ffff880028203b98] ip_rcv at ffffffff81462395
    #17 [ffff880028203bd8] __netif_receive_skb at ffffffff8142c34b
    #18 [ffff880028203c38] netif_receive_skb at ffffffff8142e408
    #19 [ffff880028203c78] napi_skb_finish at ffffffff8142e510
    #20 [ffff880028203c98] napi_gro_receive at ffffffff81430b99
    #21 [ffff880028203cb8] bnx2x_rx_int at ffffffffa02e25d6 [bnx2x]
    #22 [ffff880028203e18] bnx2x_poll at ffffffffa02e346c [bnx2x]
    #23 [ffff880028203e68] net_rx_action at ffffffff81430cb3
    #24 [ffff880028203ec8] __do_softirq at ffffffff81072191
    #25 [ffff880028203f38] call_softirq at ffffffff8100c24c
    #26 [ffff880028203f50] do_softirq at ffffffff8100de85
    #27 [ffff880028203f70] irq_exit at ffffffff81071f75
    #28 [ffff880028203f80] do_IRQ at ffffffff814f
  • Or like this one:
PID: 0      TASK: ffffffff81a8d020  CPU: 0   COMMAND: "swapper"
 #0 [ffff8800368034f0] machine_kexec at ffffffff810321cb
 #1 [ffff880036803550] crash_kexec at ffffffff810b8f22
 #2 [ffff880036803620] oops_end at ffffffff814f0560
 #3 [ffff880036803650] no_context at ffffffff8104234b
 #4 [ffff8800368036a0] __bad_area_nosemaphore at ffffffff810425d5
 #5 [ffff8800368036f0] bad_area_nosemaphore at ffffffff810426a3
 #6 [ffff880036803700] __do_page_fault at ffffffff81042d5d
 #7 [ffff880036803820] do_page_fault at ffffffff814f253e
 #8 [ffff880036803850] page_fault at ffffffff814ef8f5
    [exception RIP: tcp_mark_head_lost+185]
    RIP: ffffffff81473ad9  RSP: ffff880036803900  RFLAGS: 00010202
    RAX: 0000000000000001  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff8803ccd01c00  RDI: ffff880823f927c0
    RBP: ffff880036803930   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000002  R11: 000000003fae2a5d  R12: ffff880823f927c0
    R13: ffff88038d1d32f8  R14: 0000000000000001  R15: ffff880823f92888
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff880036803938] tcp_ack at ffffffff814798dd
#10 [ffff880036803a08] tcp_rcv_established at ffffffff8147acf8
#11 [ffff880036803a68] tcp_v4_do_rcv at ffffffff81482d23
#12 [ffff880036803b08] tcp_v4_rcv at ffffffff81484511
#13 [ffff880036803b88] ip_local_deliver_finish at ffffffff8146229d
#14 [ffff880036803bb8] ip_local_deliver at ffffffff81462528
#15 [ffff880036803be8] ip_rcv_finish at ffffffff814619ed
#16 [ffff880036803c28] ip_rcv at ffffffff81461f75
#17 [ffff880036803c68] __netif_receive_skb at ffffffff8142bf7b
#18 [ffff880036803cc8] netif_receive_skb at ffffffff8142e028
#19 [ffff880036803d08] napi_skb_finish at ffffffff8142e130
#20 [ffff880036803d28] napi_gro_receive at ffffffff814307b9
#21 [ffff880036803d48] igb_poll at ffffffffa0153d07 [igb]
#22 [ffff880036803e68] net_rx_action at ffffffff814308d3
#23 [ffff880036803ec8] __do_softirq at ffffffff81072001
#24 [ffff880036803f38] call_softirq at ffffffff8100c24c
#25 [ffff880036803f50] do_softirq at ffffffff8100de85
#26 [ffff880036803f70] irq_exit at ffffffff81071de5
#27 [ffff880036803f80] do_IRQ at ffffffff814f4dc5

Environment

  • Red Hat Enterprise Linux (RHEL) 6
    • 6.1: earlier than kernel-2.6.32-131.32.1.el6
    • 6.2: earlier than kernel-2.6.32-220.26.1.el6
    • 6.3: earlier than kernel-2.6.32-279.8.1.el6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In