RHEL6: Kernel panic in tcp_xmit_retransmit_queue() or tcp_mark_head_lost()
Issue
Server panics in tcp_xmit_retransmit_queue+128 with a trace like the following:
[exception RIP: tcp_xmit_retransmit_queue+128]
RIP: ffffffff8148c920 RSP: ffff8800282638c0 RFLAGS: 00010202
RAX: 0000000000000086 RBX: ffff8801041876c0 RCX: 0000000000000003
RDX: 000000000000002e RSI: 00000000f1a4b83d RDI: 0000000000000001
RBP: ffff880028263910 R8: 0000000000000008 R9: 00000000f1a4b83d
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffff880104187788 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#9 [ffff880028263918] tcp_fastretrans_alert at ffffffff81485f37
#10 [ffff880028263968] tcp_ack at ffffffff81486d93
#11 [ffff880028263a38] tcp_rcv_established at ffffffff81487d5d
#12 [ffff880028263a98] tcp_v4_do_rcv at ffffffff8148fd53
#13 [ffff880028263b38] tcp_v4_rcv at ffffffff814915ce
#14 [ffff880028263bb8] ip_local_deliver_finish at ffffffff8146f2ed
Or like this one:
[exception RIP: tcp_xmit_retransmit_queue+128]
RIP: ffffffff8147fca0 RSP: ffff880028203850 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880c0829b000 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff8805f89a4338 RDI: 0000000000000002
RBP: ffff8800282038a0 R8: 000000000000000d R9: 000000008b667a73
R10: 0000000000000002 R11: 000000008b6be2e9 R12: 0000000000000000
R13: 0000000000000000 R14: ffff880c0829b0c8 R15: ffff8805f89a4300
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8800282038a8] tcp_ack at ffffffff81479e03
#10 [ffff880028203978] tcp_rcv_established at ffffffff8147b138
#11 [ffff8800282039d8] tcp_v4_do_rcv at ffffffff81483163
#12 [ffff880028203a78] tcp_v4_rcv at ffffffff81484951
#13 [ffff880028203af8] ip_local_deliver_finish at ffffffff814626bd
#14 [ffff880028203b28] ip_local_deliver at ffffffff81462948
#15 [ffff880028203b58] ip_rcv_finish at ffffffff81461e0d
#16 [ffff880028203b98] ip_rcv at ffffffff81462395
#17 [ffff880028203bd8] __netif_receive_skb at ffffffff8142c34b
#18 [ffff880028203c38] netif_receive_skb at ffffffff8142e408
#19 [ffff880028203c78] napi_skb_finish at ffffffff8142e510
#20 [ffff880028203c98] napi_gro_receive at ffffffff81430b99
#21 [ffff880028203cb8] bnx2x_rx_int at ffffffffa02e25d6 [bnx2x]
#22 [ffff880028203e18] bnx2x_poll at ffffffffa02e346c [bnx2x]
#23 [ffff880028203e68] net_rx_action at ffffffff81430cb3
#24 [ffff880028203ec8] __do_softirq at ffffffff81072191
#25 [ffff880028203f38] call_softirq at ffffffff8100c24c
#26 [ffff880028203f50] do_softirq at ffffffff8100de85
#27 [ffff880028203f70] irq_exit at ffffffff81071f75
#28 [ffff880028203f80] do_IRQ at ffffffff814f
Or server panics in tcp_mark_head_lost+185 with a trace like the following:
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
#0 [ffff8800368034f0] machine_kexec at ffffffff810321cb
#1 [ffff880036803550] crash_kexec at ffffffff810b8f22
#2 [ffff880036803620] oops_end at ffffffff814f0560
#3 [ffff880036803650] no_context at ffffffff8104234b
#4 [ffff8800368036a0] __bad_area_nosemaphore at ffffffff810425d5
#5 [ffff8800368036f0] bad_area_nosemaphore at ffffffff810426a3
#6 [ffff880036803700] __do_page_fault at ffffffff81042d5d
#7 [ffff880036803820] do_page_fault at ffffffff814f253e
#8 [ffff880036803850] page_fault at ffffffff814ef8f5
[exception RIP: tcp_mark_head_lost+185]
RIP: ffffffff81473ad9 RSP: ffff880036803900 RFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8803ccd01c00 RDI: ffff880823f927c0
RBP: ffff880036803930 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000002 R11: 000000003fae2a5d R12: ffff880823f927c0
R13: ffff88038d1d32f8 R14: 0000000000000001 R15: ffff880823f92888
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff880036803938] tcp_ack at ffffffff814798dd
#10 [ffff880036803a08] tcp_rcv_established at ffffffff8147acf8
#11 [ffff880036803a68] tcp_v4_do_rcv at ffffffff81482d23
#12 [ffff880036803b08] tcp_v4_rcv at ffffffff81484511
#13 [ffff880036803b88] ip_local_deliver_finish at ffffffff8146229d
#14 [ffff880036803bb8] ip_local_deliver at ffffffff81462528
#15 [ffff880036803be8] ip_rcv_finish at ffffffff814619ed
#16 [ffff880036803c28] ip_rcv at ffffffff81461f75
#17 [ffff880036803c68] __netif_receive_skb at ffffffff8142bf7b
#18 [ffff880036803cc8] netif_receive_skb at ffffffff8142e028
#19 [ffff880036803d08] napi_skb_finish at ffffffff8142e130
#20 [ffff880036803d28] napi_gro_receive at ffffffff814307b9
#21 [ffff880036803d48] igb_poll at ffffffffa0153d07 [igb]
#22 [ffff880036803e68] net_rx_action at ffffffff814308d3
#23 [ffff880036803ec8] __do_softirq at ffffffff81072001
#24 [ffff880036803f38] call_softirq at ffffffff8100c24c
#25 [ffff880036803f50] do_softirq at ffffffff8100de85
#26 [ffff880036803f70] irq_exit at ffffffff81071de5
#27 [ffff880036803f80] do_IRQ at ffffffff814f4dc5
Environment
- Red Hat Enterprise Linux (RHEL) 6
- 6.1: earlier than kernel-2.6.32-131.32.1.el6
- 6.2: earlier than kernel-2.6.32-220.26.1.el6
- 6.3: earlier than kernel-2.6.32-279.8.1.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.