Server panics with message kernel BUG at include/linux/skbuff.h:1297!

Solution Verified - Updated -

Issue

openvswitch: panic with multiple vlan headers

------------[ cut here ]------------
kernel BUG at include/linux/skbuff.h:1297!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 15
Modules linked in: sctp openvswitch vxlan nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode power_meter acpi_ipmi ipmi_si ipmi_msghandler bnx2x libcrc32c bna cxgb3 qlge hpilo hpwdt sfc ptp pps_core mdio netxen_nic serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom qla2xxx scsi_transport_fc scsi_tgt hpsa ata_generic pata_acpi pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]

Pid: 0, comm: swapper Not tainted 2.6.32-491.el6.x86_64 #1 HP ProLiant DL585 G7
RIP: 0010:[] [] skb_pull+0x30/0x40
RSP: 0018:ffff8804474439e0 EFLAGS: 00010287
RAX: 000000000000002e RBX: ffff88063266da80 RCX: 0000000000000008
RDX: ffff88063020c6b0 RSI: 000000000000000e RDI: ffff88063266da80
RBP: ffff8804474439e0 R08: 0000000000000246 R09: ffff88063c5b16a8
R10: 860cde171500ffff R11: 1500ffffffffffff R12: ffff8806310f1020
R13: ffff8806310f1020 R14: ffff88023c007e40 R15: ffff88063266dab8
FS: 00007f526492e7c0(0000) GS:ffff880447440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000033e84ad040 CR3: 000000063bf06000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88023c97c000, task ffff88023c976aa0)
Stack:
ffff880447443a10 ffffffff8147c790 ffff880447443a50 ffff88063266da80
000000000000003c ffff8806310f1020 ffff880447443a40 ffffffffa053d399
ffff88063266da80 ffff88063020c680 ffff88063266da80 ffff88063266da80
Call Trace:

[] eth_type_trans+0x40/0x140
[] internal_dev_recv+0xc9/0x170 [openvswitch]
[] ovs_vport_send+0x22/0x90 [openvswitch]
[] ? make_writable+0x45/0x70 [openvswitch]
[] do_output+0x2e/0x50 [openvswitch]
[] do_execute_actions+0x97a/0xcd0 [openvswitch]
[] ovs_execute_actions+0x2b/0x30 [openvswitch]
[] ovs_dp_process_received_packet+0xad/0x140 [openvswitch]
[] ovs_vport_receive+0x2e/0x30 [openvswitch]
[] ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
[] __netif_receive_skb+0x60a/0x750
[] netif_receive_skb+0x58/0x60
[] napi_frags_finish+0x4f/0xb0
[] vlan_gro_frags+0xb7/0xd0
[] bnad_napi_poll_rx+0x3a9/0x910 [bna]
[] net_rx_action+0x103/0x2f0
[] __do_softirq+0xc1/0x1e0
[] ? handle_IRQ_event+0x60/0x170
[] call_softirq+0x1c/0x30
[] do_softirq+0x65/0xa0
[] irq_exit+0x85/0x90
[] do_IRQ+0x75/0xf0
[] ret_from_intr+0x0/0x11

[] ? native_safe_halt+0xb/0x10
[] default_idle+0x4d/0xb0
[] cpu_idle+0xb6/0x110
[] start_secondary+0x2be/0x301
Code: 1f 44 00 00 8b 47 68 39 f0 72 1c 29 f0 3b 47 6c 89 47 68 72 16 89 f0 48 03 87 d8 00 00 00 48 89 87 d8 00 00 00 c9 c3 31 c0 c9 c3 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53
RIP [] skb_pull+0x30/0x40
RSP

Environment

  • Red Hat Enterprise Linux (RHEL) 7.0
  • Red Hat Enterprise Linux (RHEL) 6.5
    • Seen on kernel-2.6.32-431.17.1.el6.x86_64.debug and kernel-2.6.32-431.20.3.el6.x86_64.debug
    • Does not affect systems ealier than RHEL 6.5 as the Open vSwitch (OVS) module was only introduced in 6.5.
    • OpenStack Tech Preview kernels, based on RHEL 6.4,are no longer supported, such as kernel-2.6.32-358.123.2.openstack.el6
    • Open vSwitch

Special Note
Open vSwitch and OpenStack related capabilities have been added into RHEL 6.5 to provide feature enablement for Red Hat OpenStack Platform (RH-OSP) and are only supported with RH-OSP.

While a Red Hat OpenStack Preview used a modified kernel based on RHEL 6.4 to include various feature enablement for Open vSwitch and OpenStack technologies, these features are only tested and supported on the Red Hat Enterprise Linux OpenStack Platform. This special RHEL 6.4 kernel-2.6.32-358.123.2.openstack.el6 was intended only as a Tech Preview and is no longer supported by Red Hat engineering.

This is further explained in RHEL 6.5 Technical Release Notes:

Open vSwitch (OVS) is an open-source, multi-layer software switch designed to be used as a virtual switch in virtualized server environments. Starting with Red Hat Enterprise Linux 6.4, the Open vSwitch kernel module is included as an enabler for Red Hat Enterprise Linux OpenStack Platform. Open vSwitch is only supported in conjunction with Red Hat products containing the accompanying user-space packages. Without theses packages, Open vSwitch will not function and cannot be used with other Red Hat Enterprise Linux variants.

The various OpenStack community platforms, including the RDO distribution of the OpenStack, is not supported by Red Hat Global Support Services.
Please refer to Is the RDO distribution of OpenStack supported by Red Hat?.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content