Random reboot with "kernel BUG at net/core/skbuff.c" or "skb_over_panic" in the transmit path
Issue
Machines are randomly rebooting/panicking with the following in the vmcore-dmesg.txt and back trace:
<2>[259117.681597] kernel BUG at net/core/skbuff.c:1886!
<4>[259117.738957] invalid opcode: 0000 [#1] SMP
<4>[259117.789236] last sysfs file: /sys/devices/system/cpu/online
<4>[259117.857008] CPU 0
crash> bt
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
#0 [ffff880061a03190] machine_kexec at ffffffff8103b60b
#1 [ffff880061a031f0] crash_kexec at ffffffff810c99e2
#2 [ffff880061a032c0] oops_end at ffffffff8152e0d0
#3 [ffff880061a032f0] die at ffffffff81010f5b
#4 [ffff880061a03320] do_trap at ffffffff8152d934
#5 [ffff880061a03380] do_invalid_op at ffffffff8100cf55
#6 [ffff880061a03420] invalid_op at ffffffff8100c01b
[exception RIP: skb_checksum+725]
RIP: ffffffff81451ff5 RSP: ffff880061a034d0 RFLAGS: 00010206
RAX: ffff881062d2dd80 RBX: 0000000000000090 RCX: 00000000ff2a5251
RDX: 0000000000000180 RSI: 0000000000000000 RDI: ffff881062d2dd40
RBP: ffff880061a03530 R8: ffff880fe9624868 R9: ffff881062d2dd40
R10: 0000000000000020 R11: 0000000000000003 R12: 0000000000000042
R13: 0000000000000042 R14: 0000000000000000 R15: ffffffffa014a580
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff880061a03538] skb_checksum_help at ffffffff8145d60a
#8 [ffff880061a03568] dev_hard_start_xmit at ffffffff81461f8d
#9 [ffff880061a035c8] sch_direct_xmit at ffffffff8147ec3a
#10 [ffff880061a03618] dev_queue_xmit at ffffffff81462368
#11 [ffff880061a03658] bond_dev_queue_xmit at ffffffffa024b93c [bonding]
#12 [ffff880061a03668] bond_start_xmit at ffffffffa024bc51 [bonding]
#13 [ffff880061a036b8] dev_hard_start_xmit at ffffffff81461e5c
#14 [ffff880061a03718] dev_queue_xmit at ffffffff814622fd
#15 [ffff880061a03758] ip_finish_output at ffffffff8149d618
#16 [ffff880061a037a8] ip_output at ffffffff8149d898
#17 [ffff880061a037d8] ip_local_out at ffffffff8149cb75
#18 [ffff880061a037f8] ip_queue_xmit at ffffffff8149d070
#19 [ffff880061a038a8] tcp_transmit_skb at ffffffff814b25a4
#20 [ffff880061a03928] tcp_write_xmit at ffffffff814b4aea
#21 [ffff880061a039b8] __tcp_push_pending_frames at ffffffff814b56d0
#22 [ffff880061a039d8] tcp_data_snd_check at ffffffff814accd3
#23 [ffff880061a039f8] tcp_rcv_established at ffffffff814b0941
#24 [ffff880061a03a58] tcp_v4_do_rcv at ffffffff814b8e13
#25 [ffff880061a03af8] tcp_v4_rcv at ffffffff814ba732
#26 [ffff880061a03b88] ip_local_deliver_finish at ffffffff814974ed
#27 [ffff880061a03bb8] ip_local_deliver at ffffffff81497778
#28 [ffff880061a03be8] ip_rcv_finish at ffffffff81496c3d
#29 [ffff880061a03c28] ip_rcv at ffffffff814971c5
#30 [ffff880061a03c68] __netif_receive_skb at ffffffff8145cdb8
#31 [ffff880061a03cb8] netif_receive_skb at ffffffff81460b78
#32 [ffff880061a03cf8] napi_skb_finish at ffffffff81460c80
#33 [ffff880061a03d18] napi_gro_receive at ffffffff81462539
#34 [ffff880061a03d38] ixgbe_clean_rx_irq at ffffffffa011bcca [ixgbe]
#35 [ffff880061a03db8] ixgbe_poll at ffffffffa0121583 [ixgbe]
#36 [ffff880061a03e68] net_rx_action at ffffffff81462653
#37 [ffff880061a03ec8] __do_softirq at ffffffff8107d901
#38 [ffff880061a03f38] call_softirq at ffffffff8100c38c
#39 [ffff880061a03f50] do_softirq at ffffffff8100fbd5
#40 [ffff880061a03f70] irq_exit at ffffffff8107d7b5
#41 [ffff880061a03f80] do_IRQ at ffffffff81533ba5
RIP: 0010:[<ffffffff8146b2fd>] [<ffffffff8146b2fd>] skb_over_panic+0x5d/0x70
...
Call Trace:
[<ffffffff814c2ee2>] ? sk_stream_alloc_skb+0xb2/0x110
[<ffffffff8146c0ec>] skb_put+0x5c/0x70
[<ffffffff814c348b>] tcp_sendmsg+0x54b/0xa20
[<ffffffff81464843>] sock_sendmsg+0x123/0x150
Environment
- Potentially all RHEL6 versions, seen at least on:
- Red Hat Enterprise Linux 6.6 (
kernel-2.6.32-504.23.4.el6) - Red Hat Enterprise Linux 6.8 (
2.6.32-642.4.2.el6)
- Red Hat Enterprise Linux 6.6 (
- Network interface with offloading enabled
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
