Random reboot with "kernel BUG at net/core/skbuff.c" or "skb_over_panic" in the transmit path
Issue
Machines are randomly rebooting/panicking with the following in the vmcore-dmesg.txt and back trace:
<2>[259117.681597] kernel BUG at net/core/skbuff.c:1886!
<4>[259117.738957] invalid opcode: 0000 [#1] SMP
<4>[259117.789236] last sysfs file: /sys/devices/system/cpu/online
<4>[259117.857008] CPU 0
crash> bt
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
#0 [ffff880061a03190] machine_kexec at ffffffff8103b60b
#1 [ffff880061a031f0] crash_kexec at ffffffff810c99e2
#2 [ffff880061a032c0] oops_end at ffffffff8152e0d0
#3 [ffff880061a032f0] die at ffffffff81010f5b
#4 [ffff880061a03320] do_trap at ffffffff8152d934
#5 [ffff880061a03380] do_invalid_op at ffffffff8100cf55
#6 [ffff880061a03420] invalid_op at ffffffff8100c01b
[exception RIP: skb_checksum+725]
RIP: ffffffff81451ff5 RSP: ffff880061a034d0 RFLAGS: 00010206
RAX: ffff881062d2dd80 RBX: 0000000000000090 RCX: 00000000ff2a5251
RDX: 0000000000000180 RSI: 0000000000000000 RDI: ffff881062d2dd40
RBP: ffff880061a03530 R8: ffff880fe9624868 R9: ffff881062d2dd40
R10: 0000000000000020 R11: 0000000000000003 R12: 0000000000000042
R13: 0000000000000042 R14: 0000000000000000 R15: ffffffffa014a580
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff880061a03538] skb_checksum_help at ffffffff8145d60a
#8 [ffff880061a03568] dev_hard_start_xmit at ffffffff81461f8d
#9 [ffff880061a035c8] sch_direct_xmit at ffffffff8147ec3a
#10 [ffff880061a03618] dev_queue_xmit at ffffffff81462368
#11 [ffff880061a03658] bond_dev_queue_xmit at ffffffffa024b93c [bonding]
#12 [ffff880061a03668] bond_start_xmit at ffffffffa024bc51 [bonding]
#13 [ffff880061a036b8] dev_hard_start_xmit at ffffffff81461e5c
#14 [ffff880061a03718] dev_queue_xmit at ffffffff814622fd
#15 [ffff880061a03758] ip_finish_output at ffffffff8149d618
#16 [ffff880061a037a8] ip_output at ffffffff8149d898
#17 [ffff880061a037d8] ip_local_out at ffffffff8149cb75
#18 [ffff880061a037f8] ip_queue_xmit at ffffffff8149d070
#19 [ffff880061a038a8] tcp_transmit_skb at ffffffff814b25a4
#20 [ffff880061a03928] tcp_write_xmit at ffffffff814b4aea
#21 [ffff880061a039b8] __tcp_push_pending_frames at ffffffff814b56d0
#22 [ffff880061a039d8] tcp_data_snd_check at ffffffff814accd3
#23 [ffff880061a039f8] tcp_rcv_established at ffffffff814b0941
#24 [ffff880061a03a58] tcp_v4_do_rcv at ffffffff814b8e13
#25 [ffff880061a03af8] tcp_v4_rcv at ffffffff814ba732
#26 [ffff880061a03b88] ip_local_deliver_finish at ffffffff814974ed
#27 [ffff880061a03bb8] ip_local_deliver at ffffffff81497778
#28 [ffff880061a03be8] ip_rcv_finish at ffffffff81496c3d
#29 [ffff880061a03c28] ip_rcv at ffffffff814971c5
#30 [ffff880061a03c68] __netif_receive_skb at ffffffff8145cdb8
#31 [ffff880061a03cb8] netif_receive_skb at ffffffff81460b78
#32 [ffff880061a03cf8] napi_skb_finish at ffffffff81460c80
#33 [ffff880061a03d18] napi_gro_receive at ffffffff81462539
#34 [ffff880061a03d38] ixgbe_clean_rx_irq at ffffffffa011bcca [ixgbe]
#35 [ffff880061a03db8] ixgbe_poll at ffffffffa0121583 [ixgbe]
#36 [ffff880061a03e68] net_rx_action at ffffffff81462653
#37 [ffff880061a03ec8] __do_softirq at ffffffff8107d901
#38 [ffff880061a03f38] call_softirq at ffffffff8100c38c
#39 [ffff880061a03f50] do_softirq at ffffffff8100fbd5
#40 [ffff880061a03f70] irq_exit at ffffffff8107d7b5
#41 [ffff880061a03f80] do_IRQ at ffffffff81533ba5
RIP: 0010:[<ffffffff8146b2fd>] [<ffffffff8146b2fd>] skb_over_panic+0x5d/0x70
...
Call Trace:
[<ffffffff814c2ee2>] ? sk_stream_alloc_skb+0xb2/0x110
[<ffffffff8146c0ec>] skb_put+0x5c/0x70
[<ffffffff814c348b>] tcp_sendmsg+0x54b/0xa20
[<ffffffff81464843>] sock_sendmsg+0x123/0x150
Environment
- Potentially all RHEL6 versions, seen at least on:
- Red Hat Enterprise Linux 6.6 (
kernel-2.6.32-504.23.4.el6) - Red Hat Enterprise Linux 6.8 (
2.6.32-642.4.2.el6)
- Red Hat Enterprise Linux 6.6 (
- Network interface with offloading enabled
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.