kernel BUG at net/core/skbuff.c:2684 due to vlan regression in Red Hat Enterprise Linux

Solution Verified - Updated -

Issue

Experienced a crash of prod server due to a possible regression in VLAN and GRO handling code.
* Check vmcore for the following signature.

------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:2684!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu31/topology/thread_siblings
CPU 7 
Modules linked in: xt_multiport nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac xt_state xt_physdev iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bridge veth ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables cpufreq_ondemand freq_table pcc_cpufreq 8021q garp stp llc ipv6 openvswitch vxlan vhost_net macvtap macvlan tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode hpilo hpwdt igb i2c_algo_bit i2c_core ptp pps_core power_meter sg bnx2x libcrc32c mdio serio_raw lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif hpsa dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 0, comm: swapper Not tainted 2.6.32-431.el6.x86_64 #1 HP ProLiant SL4540 Gen8 /
RIP: 0010:[<ffffffff81452d3f>]  [<ffffffff81452d3f>] skb_segment+0x76f/0x7a0
RSP: 0018:ffff8801000e3540  EFLAGS: 00010216
RAX: 0000000000000000 RBX: ffff881604c48880 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8815bb8de884 RDI: ffff88166fb91884
RBP: ffff8801000e3600 R08: 00000000000005ee R09: ffff88161f146780
R10: 0000000000000218 R11: 0000000000000000 R12: ffff881607a6abc0
R13: 00000000000005ee R14: ffff88161f146780 R15: 00000000000005a8
FS:  0000000000000000(0000) GS:ffff8801000e0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fea220b3a08 CR3: 00000018577c1000 CR4: 00000000000427e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88185b1f4000, task ffff88305b1c6080)
Stack:
 ffff8801000e3590 0000000700000006 ffffffff00000000 0000000000000000
<d> 000000000000003e 0000003e00000000 0100000000000000 0000000000000000
<d> 0000000000000046 ffffffffffffffba 0000000000000000 ffff881604c48880
Call Trace:
 <IRQ> 
 [<ffffffff814a38f4>] tcp_tso_segment+0xf4/0x300
 [<ffffffff81059216>] ? enqueue_task+0x66/0x80
 [<ffffffff814c7761>] inet_gso_segment+0x101/0x2b0
 [<ffffffff8145fe82>] skb_mac_gso_segment+0xa2/0x270
 [<ffffffff814600a7>] __skb_gso_segment+0x57/0xc0
 [<ffffffff81460123>] skb_gso_segment+0x13/0x20
 [<ffffffff814601cb>] dev_hard_start_xmit+0x9b/0x480
 [<ffffffff8147bd0a>] sch_direct_xmit+0x15a/0x1c0
 [<ffffffff81460858>] dev_queue_xmit+0x228/0x320
 [<ffffffffa0364898>] br_dev_queue_push_xmit+0x88/0xc0 [bridge]
 [<ffffffffa036a768>] br_nf_dev_queue_xmit+0x28/0xa0 [bridge]
 [<ffffffffa036b480>] br_nf_post_routing+0x1d0/0x280 [bridge]
 [<ffffffff814893a9>] nf_iterate+0x69/0xb0
 [<ffffffffa0364810>] ? br_dev_queue_push_xmit+0x0/0xc0 [bridge]
 [<ffffffff81489566>] nf_hook_slow+0x76/0x120
 [<ffffffffa0364810>] ? br_dev_queue_push_xmit+0x0/0xc0 [bridge]
 [<ffffffffa03648d0>] ? br_forward_finish+0x0/0x60 [bridge]
 [<ffffffffa0364913>] br_forward_finish+0x43/0x60 [bridge]
 [<ffffffffa036acc8>] br_nf_forward_finish+0x128/0x140 [bridge]
 [<ffffffffa036b248>] br_nf_forward_ip+0x2f8/0x360 [bridge]
 [<ffffffff814893a9>] nf_iterate+0x69/0xb0
 [<ffffffffa03648d0>] ? br_forward_finish+0x0/0x60 [bridge]
 [<ffffffff81489566>] nf_hook_slow+0x76/0x120
 [<ffffffffa03648d0>] ? br_forward_finish+0x0/0x60 [bridge]
 [<ffffffffa0364930>] ? __br_forward+0x0/0xd0 [bridge]
 [<ffffffffa03649ae>] __br_forward+0x7e/0xd0 [bridge]
 [<ffffffffa03647c1>] br_flood+0xc1/0xd0 [bridge]
 [<ffffffffa03647e5>] br_flood_forward+0x15/0x20 [bridge]
 [<ffffffffa0365b6e>] br_handle_frame_finish+0x27e/0x2a0 [bridge]
 [<ffffffffa036b938>] br_nf_pre_routing_finish+0x238/0x350 [bridge]
 [<ffffffffa036bedb>] br_nf_pre_routing+0x48b/0x7b0 [bridge]
 [<ffffffff814893a9>] nf_iterate+0x69/0xb0
 [<ffffffffa03658f0>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]
 [<ffffffff81489566>] nf_hook_slow+0x76/0x120
 [<ffffffffa03658f0>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]
 [<ffffffffa0365d1c>] br_handle_frame+0x18c/0x250 [bridge]
 [<ffffffffa02de513>] ? ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
 [<ffffffff8145b5c9>] __netif_receive_skb+0x529/0x750
 [<ffffffff8145b88a>] process_backlog+0x9a/0x100
 [<ffffffff81460b43>] net_rx_action+0x103/0x2f0
 [<ffffffff8107a8e1>] __do_softirq+0xc1/0x1e0
 [<ffffffff810e6ec0>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8100c30c>] call_softirq+0x1c/0x30
 [<ffffffff8100fa75>] do_softirq+0x65/0xa0
 [<ffffffff8107a795>] irq_exit+0x85/0x90
 [<ffffffff81530fe5>] do_IRQ+0x75/0xf0
 [<ffffffff8100b9d3>] ret_from_intr+0x0/0x11
 <EOI> 
 [<ffffffff812e09be>] ? intel_idle+0xde/0x170
 [<ffffffff812e09a1>] ? intel_idle+0xc1/0x170
 [<ffffffff814266f7>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009fc6>] cpu_idle+0xb6/0x110
 [<ffffffff81520e13>] start_secondary+0x2ac/0x2ef
Code: 06 0f 1f 00 48 89 df 48 8b 1f e8 ad cb ff ff 48 85 db 75 f0 48 c7 45 98 f4 ff ff ff e9 e5 fc ff ff f0 41 ff 86 e4 00 00 00 eb 97 <0f> 0b eb fe 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 eb f3 0f 0b 
RIP  [<ffffffff81452d3f>] skb_segment+0x76f/0x7a0
 RSP <ffff8801000e3540>

Environment

  • Red Hat Enterprise Linux (RHEL) 6.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In