RHEL6: open vswitch process ovs-vswitchd crashed in the kernel cache_alloc_refill, corruption in either the skbuff_fclone_cache or skbuff_head_cache
Issue
- process ovs-vswitchd crashed in the kernel in cache_alloc_refill, kernel BUG at mm/slab.c:3069
<2>kernel BUG at mm/slab.c:3069!
<4>invalid opcode: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/system/node/node1/meminfo
<4>CPU 0
<4>Modules linked in: openvswitch(U) ip6table_filter ip6_tables ebtable_nat ebtables nf_defrag_ipv4
ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables vhost_net macvtap macvlan tun kvm_intel
kvm sunrpc bonding 8021q garp stp llc ipv6 power_meter hpilo(U) hpwdt igb ixgbe ptp pps_core mdio
sg serio_raw iTCO_wdt iTCO_vendor_support ioatdma dca shpchp ext4 mbcache jbd2 sd_mod crc_t10dif
hpvsa(P)(U) ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_conntrack]
<4>
<4>Pid: 37895, comm: ovs-vswitchd Tainted: P --------------- 2.6.32-358.el6.x86_64 #1 HP ProLiant SL230s Gen8/
<4>RIP: 0010:[<ffffffff81167274>] [<ffffffff81167274>] cache_alloc_refill+0x1e4/0x240
<4>RSP: 0000:ffff880123c03ab0 EFLAGS: 00010046
<4>RAX: 000000000000000d RBX: ffff884052ce0540 RCX: 000000000000000f
<4>RDX: ffff880bed160000 RSI: ffff882052c66140 RDI: ffff8816bbcfd000
<4>RBP: ffff880123c03b10 R08: ffff880bed160000 R09: ffff8804f5be9680
<4>R10: 0000000000000070 R11: 0000000000000000 R12: ffff882052e52400
<4>R13: ffff882052c66140 R14: 000000000000000d R15: ffff8816bbcfd000
<4>FS: 00007f437068e7c0(0000) GS:ffff880123c00000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 00007f9e70020000 CR3: 00000040504a6000 CR4: 00000000000427e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process ovs-vswitchd (pid: 37895, threadinfo ffff88404a18c000, task ffff88404abd0aa0)
<4>Stack:
<4> cb4f0f0200000000 000000008147f1b0 ffff882052c66180 0004122044396480
<4><d> ffff882052c66160 ffff882052c66150 ffff880123c03b60 0000000000000000
<4><d> 0000000000000020 00000000ffffffff ffff884052ce0540 0000000000000020
<4>Call Trace:
<4> <IRQ>
<4> [<ffffffff81166d5e>] kmem_cache_alloc_node+0x1be/0x1d0
<4> [<ffffffff8143d68f>] __alloc_skb+0x4f/0x190
<4> [<ffffffff8143d834>] __netdev_alloc_skb+0x24/0x50
<4> [<ffffffff8143da01>] skb_gro_receive+0x1a1/0x4b0
<4> [<ffffffff812771e9>] ? cpumask_next_and+0x29/0x50
<4> [<ffffffff8148e2e1>] tcp_gro_receive+0x271/0x2d0
<4> [<ffffffff8149e7ba>] tcp4_gro_receive+0x5a/0xd0
<4> [<ffffffff814b17cd>] inet_gro_receive+0x17d/0x220
<4> [<ffffffff8144a950>] dev_gro_receive+0x1d0/0x2d0
<4> [<ffffffff8144ab4b>] __napi_gro_receive+0xfb/0x140
<4> [<ffffffff8144cc1f>] napi_gro_receive+0x2f/0x50
<4> [<ffffffffa039344c>] ixgbe_poll+0xa6c/0x1280 [ixgbe]
<4> [<ffffffff8144cd43>] net_rx_action+0x103/0x2f0
<4> [<ffffffff81076fb1>] __do_softirq+0xc1/0x1e0
<4> [<ffffffff810e1640>] ? handle_IRQ_event+0x60/0x170
<4> [<ffffffff8100c1cc>] call_softirq+0x1c/0x30
<4> [<ffffffff8100de05>] do_softirq+0x65/0xa0
<4> [<ffffffff81076d95>] irq_exit+0x85/0x90
<4> [<ffffffff81516c95>] do_IRQ+0x75/0xf0
<4> [<ffffffff8100b9d3>] ret_from_intr+0x0/0x11
<4> <EOI>
<4>Code: 89 ff e8 c0 1b 12 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31
<1>RIP [<ffffffff81167274>] cache_alloc_refill+0x1e4/0x240
<4> RSP <ffff880123c03ab0>
Environment
- Red Hat Enterprise Linux 6
- seen on kernel 2.6.32-358.el6.x86_64
- The following 3rd party modules are seen loaded.
openvswitch(U) ... hpilo(U) ... hpvsa(P)(U)
- In particular, the 'openvswitch' module is of particular interest due to the fact that:
- The process which crashed in the kernel was ovs-vswitchd, which is an openvswitch process
- This is a 3rd party module, related to networking and the cause of the crash was a networking slab cache corruption.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
