kernel BUG at include/linux/skbuff.h:1245!

Solution Verified - Updated -

Environment

  • Openstack
  • kernel-2.6.32-431.27.1.el6.x86_64

Issue

Kernel panic on Openstack using openvswitch module

Resolution

  • This issue was being tracked in Red Hat Bugzilla 1122405, and was resolved in 2.6.32-504.el6 (RHEL6.6GA) or later with the following errata:RHSA-2014-1392. Note: This Bug is not publicly accessible and you should contact your Red Hat Support representative if more information is required).

Root Cause

  • The openvswitch driver did not handle frames that contained multiple VLAN headers correctly, which could result in a kernel panic.

Diagnostic Steps

  • Check vmcore for the folowing signature.
------------[ cut here ]------------
kernel BUG at include/linux/skbuff.h:1245!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 33 
Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat xt_mac nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_physdev xt_multiport bridge veth ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables sunrpc bonding ipv6 8021q garp stp llc openvswitch vxlan vhost_net macvtap macvlan tun kvm_amd kvm tpm_infineon microcode power_meter sg hpilo hpwdt be2iscsi iscsi_boot_sysfs libiscsi scsi_transport_iscsi be2net serio_raw fam15h_power k10temp i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif hpsa ata_generic pata_acpi pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]

Pid: 0, comm: swapper Not tainted 2.6.32-431.27.1.el6.x86_64 #1 HP ProLiant BL685c G7
RIP: 0010:[<ffffffff8144f540>]  [<ffffffff8144f540>] skb_pull+0x30/0x40
RSP: 0018:ffff8860b0c039b0  EFLAGS: 00010297
RAX: 0000000000000034 RBX: ffff887ff9641bc0 RCX: 0000000000000000
RDX: 00000000000005dc RSI: 000000000000000e RDI: ffff887ff9641bc0
RBP: ffff8860b0c039b0 R08: 0000000000005500 R09: 0000000000000000
R10: cfccf93e16fa1583 R11: 16fa15839e3e16fa R12: ffff8820125e5020
R13: 0000000000000042 R14: ffff88200d371cc0 R15: ffffffffa00328e0
FS:  00007f2ce76b77c0(0000) GS:ffff8860b0c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fea178652d8 CR3: 000000c0018cc000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff888013e44000, task ffff886013ef7540)
Stack:
 ffff8860b0c039e0 ffffffff8147b680 ffff887f4b769a00 ffff887ff9641bc0
<d> ffff8820125e5020 0000000000000042 ffff8860b0c03a00 ffffffff81461e81
<d> ffffe8800b03fed0 ffffe8800b03dc20 ffff8860b0c03a30 ffffffffa0032770
Call Trace:
 <IRQ> 
 [<ffffffff8147b680>] eth_type_trans+0x40/0x140
 [<ffffffff81461e81>] dev_forward_skb+0x101/0x210
 [<ffffffffa0032770>] veth_xmit+0x80/0xf0 [veth]
 [<ffffffff81461104>] dev_hard_start_xmit+0x224/0x480
 [<ffffffffa03a4fe2>] ? ovs_vport_send+0x22/0x90 [openvswitch]
 [<ffffffff8147cb9a>] sch_direct_xmit+0x15a/0x1c0
 [<ffffffffa044c8f0>] ? br_forward_finish+0x0/0x60 [bridge]
 [<ffffffff81461608>] dev_queue_xmit+0x228/0x320
 [<ffffffffa03a621d>] netdev_send+0x4d/0xc0 [openvswitch]
 [<ffffffffa03a4fe2>] ovs_vport_send+0x22/0x90 [openvswitch]
 [<ffffffffa03a0a3c>] ? ovs_dp_process_received_packet+0xac/0x130 [openvswitch]
 [<ffffffffa039e3fe>] do_output+0x2e/0x50 [openvswitch]
 [<ffffffffa039ecb6>] do_execute_actions+0x896/0xbd0 [openvswitch]
 [<ffffffff81292b92>] ? flex_array_get+0x12/0x80
 [<ffffffffa03a3b46>] ? find_bucket+0x66/0x70 [openvswitch]
 [<ffffffffa039f01b>] ovs_execute_actions+0x2b/0x30 [openvswitch]
 [<ffffffffa03a0a3c>] ovs_dp_process_received_packet+0xac/0x130 [openvswitch]
 [<ffffffffa03a6513>] ? ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
 [<ffffffffa03a507e>] ovs_vport_receive+0x2e/0x30 [openvswitch]
 [<ffffffffa03a6513>] ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
 [<ffffffff8145c45a>] __netif_receive_skb+0x60a/0x750
 [<ffffffff8145c63a>] process_backlog+0x9a/0x100
 [<ffffffff814618f3>] net_rx_action+0x103/0x2f0
 [<ffffffff8107a5f1>] __do_softirq+0xc1/0x1e0
 [<ffffffff810e6d40>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff8100c30c>] call_softirq+0x1c/0x30
 [<ffffffff8100fa75>] do_softirq+0x65/0xa0
 [<ffffffff8107a4a5>] irq_exit+0x85/0x90
 [<ffffffff815322d5>] do_IRQ+0x75/0xf0
 [<ffffffff8100b9d3>] ret_from_intr+0x0/0x11
 <EOI> 
 [<ffffffff81315ea4>] ? acpi_idle_enter_simple+0x114/0x14b
 [<ffffffff81315ea0>] ? acpi_idle_enter_simple+0x110/0x14b
 [<ffffffff81427477>] cpuidle_idle_call+0xa7/0x140
 [<ffffffff81009fc6>] cpu_idle+0xb6/0x110
 [<ffffffff8152202c>] start_secondary+0x2ac/0x2ef
Code: 1f 44 00 00 8b 47 68 39 f0 72 1c 29 f0 3b 47 6c 89 47 68 72 16 89 f0 48 03 87 d8 00 00 00 48 89 87 d8 00 00 00 c9 c3 31 c0 c9 c3 <0f> 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 
RIP  [<ffffffff8144f540>] skb_pull+0x30/0x40
 RSP <ffff8860b0c039b0>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments