RHEL6: Kernel panic in tcp_xmit_retransmit_queue() or tcp_mark_head_lost
Issue
- Server panics in tcp_xmit_retransmit_queue+128 with a trace like the following:
[exception RIP: tcp_xmit_retransmit_queue+128]
RIP: ffffffff8148c920 RSP: ffff8800282638c0 RFLAGS: 00010202
RAX: 0000000000000086 RBX: ffff8801041876c0 RCX: 0000000000000003
RDX: 000000000000002e RSI: 00000000f1a4b83d RDI: 0000000000000001
RBP: ffff880028263910 R8: 0000000000000008 R9: 00000000f1a4b83d
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffff880104187788 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
#9 [ffff880028263918] tcp_fastretrans_alert at ffffffff81485f37
#10 [ffff880028263968] tcp_ack at ffffffff81486d93
#11 [ffff880028263a38] tcp_rcv_established at ffffffff81487d5d
#12 [ffff880028263a98] tcp_v4_do_rcv at ffffffff8148fd53
#13 [ffff880028263b38] tcp_v4_rcv at ffffffff814915ce
#14 [ffff880028263bb8] ip_local_deliver_finish at ffffffff8146f2ed
- Or like this one:
[exception RIP: tcp_xmit_retransmit_queue+128]
RIP: ffffffff8147fca0 RSP: ffff880028203850 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880c0829b000 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffff8805f89a4338 RDI: 0000000000000002
RBP: ffff8800282038a0 R8: 000000000000000d R9: 000000008b667a73
R10: 0000000000000002 R11: 000000008b6be2e9 R12: 0000000000000000
R13: 0000000000000000 R14: ffff880c0829b0c8 R15: ffff8805f89a4300
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8800282038a8] tcp_ack at ffffffff81479e03
#10 [ffff880028203978] tcp_rcv_established at ffffffff8147b138
#11 [ffff8800282039d8] tcp_v4_do_rcv at ffffffff81483163
#12 [ffff880028203a78] tcp_v4_rcv at ffffffff81484951
#13 [ffff880028203af8] ip_local_deliver_finish at ffffffff814626bd
#14 [ffff880028203b28] ip_local_deliver at ffffffff81462948
#15 [ffff880028203b58] ip_rcv_finish at ffffffff81461e0d
#16 [ffff880028203b98] ip_rcv at ffffffff81462395
#17 [ffff880028203bd8] __netif_receive_skb at ffffffff8142c34b
#18 [ffff880028203c38] netif_receive_skb at ffffffff8142e408
#19 [ffff880028203c78] napi_skb_finish at ffffffff8142e510
#20 [ffff880028203c98] napi_gro_receive at ffffffff81430b99
#21 [ffff880028203cb8] bnx2x_rx_int at ffffffffa02e25d6 [bnx2x]
#22 [ffff880028203e18] bnx2x_poll at ffffffffa02e346c [bnx2x]
#23 [ffff880028203e68] net_rx_action at ffffffff81430cb3
#24 [ffff880028203ec8] __do_softirq at ffffffff81072191
#25 [ffff880028203f38] call_softirq at ffffffff8100c24c
#26 [ffff880028203f50] do_softirq at ffffffff8100de85
#27 [ffff880028203f70] irq_exit at ffffffff81071f75
#28 [ffff880028203f80] do_IRQ at ffffffff814f
- Or like this one:
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
#0 [ffff8800368034f0] machine_kexec at ffffffff810321cb
#1 [ffff880036803550] crash_kexec at ffffffff810b8f22
#2 [ffff880036803620] oops_end at ffffffff814f0560
#3 [ffff880036803650] no_context at ffffffff8104234b
#4 [ffff8800368036a0] __bad_area_nosemaphore at ffffffff810425d5
#5 [ffff8800368036f0] bad_area_nosemaphore at ffffffff810426a3
#6 [ffff880036803700] __do_page_fault at ffffffff81042d5d
#7 [ffff880036803820] do_page_fault at ffffffff814f253e
#8 [ffff880036803850] page_fault at ffffffff814ef8f5
[exception RIP: tcp_mark_head_lost+185]
RIP: ffffffff81473ad9 RSP: ffff880036803900 RFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8803ccd01c00 RDI: ffff880823f927c0
RBP: ffff880036803930 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000002 R11: 000000003fae2a5d R12: ffff880823f927c0
R13: ffff88038d1d32f8 R14: 0000000000000001 R15: ffff880823f92888
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff880036803938] tcp_ack at ffffffff814798dd
#10 [ffff880036803a08] tcp_rcv_established at ffffffff8147acf8
#11 [ffff880036803a68] tcp_v4_do_rcv at ffffffff81482d23
#12 [ffff880036803b08] tcp_v4_rcv at ffffffff81484511
#13 [ffff880036803b88] ip_local_deliver_finish at ffffffff8146229d
#14 [ffff880036803bb8] ip_local_deliver at ffffffff81462528
#15 [ffff880036803be8] ip_rcv_finish at ffffffff814619ed
#16 [ffff880036803c28] ip_rcv at ffffffff81461f75
#17 [ffff880036803c68] __netif_receive_skb at ffffffff8142bf7b
#18 [ffff880036803cc8] netif_receive_skb at ffffffff8142e028
#19 [ffff880036803d08] napi_skb_finish at ffffffff8142e130
#20 [ffff880036803d28] napi_gro_receive at ffffffff814307b9
#21 [ffff880036803d48] igb_poll at ffffffffa0153d07 [igb]
#22 [ffff880036803e68] net_rx_action at ffffffff814308d3
#23 [ffff880036803ec8] __do_softirq at ffffffff81072001
#24 [ffff880036803f38] call_softirq at ffffffff8100c24c
#25 [ffff880036803f50] do_softirq at ffffffff8100de85
#26 [ffff880036803f70] irq_exit at ffffffff81071de5
#27 [ffff880036803f80] do_IRQ at ffffffff814f4dc5
Environment
- Red Hat Enterprise Linux (RHEL) 6
- 6.1: earlier than kernel-2.6.32-131.32.1.el6
- 6.2: earlier than kernel-2.6.32-220.26.1.el6
- 6.3: earlier than kernel-2.6.32-279.8.1.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
