Kernel panic of network node when guests access network

Solution Verified - Updated -

Issue

Network Node panics with the following back trace when load is applied on the Tenant networks

[ 1088.309799] ------------[ cut here ]------------
[ 1088.309893] kernel BUG at net/core/skbuff.c:2979!
[ 1088.309923] invalid opcode: 0000 [#1] SMP 
[ 1088.309954] Modules linked in: ip6t_rpfilter ip6t_REJECT ebtable_nat ebtable_broute ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ip6table_filter ip6_tables xt_nat xt_conntrack xt_REDIRECT veth 8021q garp mrp ipt_REJECT bridge openvswitch stp vxlan llc ip_tunnel gre libcrc32c iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables sg iTCO_wdt ipmi_devintf iTCO_vendor_support dcdbas coretemp kvm_intel kvm crc32c_intel pcspkr lpc_ich mfd_core ipmi_si ipmi_msghandler bnx2 i7core_edac edac_core acpi_power_meter shpchp nfsd mperf auth_rpcgss nfs_acl lockd sunrpc ext4 mbcache jbd2 dm_service_time sr_mod cdrom sd_mod crc_t10dif crct10dif_common
[ 1088.310573]  usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm qla2xxx i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_multipath dm_mod [last unloaded: iptable_raw]
[ 1088.310734] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G          I --------------   3.10.0-123.6.3.el7.x86_64 #1
[ 1088.310792] Hardware name: Dell Inc. PowerEdge M610/0N582M, BIOS 1.3.6 10/30/2009
[ 1088.310837] task: ffff8817ece90b60 ti: ffff8817ece8e000 task.ti: ffff8817ece8e000
[ 1088.310881] RIP: 0010:[<ffffffff814c413a>]  [<ffffffff814c413a>] skb_segment+0x93a/0x960
[ 1088.310935] RSP: 0018:ffff880bffce3478  EFLAGS: 00010202
[ 1088.310968] RAX: 0000000000000000 RBX: ffff880beca94300 RCX: ffff880be8d3a0f0
[ 1088.311010] RDX: 0000000000000050 RSI: ffff880be8d3a000 RDI: ffff880be8d3b200
[ 1088.311052] RBP: ffff880bffce3530 R08: 0000000000000042 R09: 0000000000000000
[ 1088.311094] R10: ffff880beca94600 R11: 000000000000057c R12: ffff880be8d3b2f0
[ 1088.311136] R13: 0000000000000000 R14: ffff880beca94500 R15: 0000000000000050
[ 1088.311178] FS:  0000000000000000(0000) GS:ffff880bffce0000(0000) knlGS:0000000000000000
[ 1088.311226] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1088.311260] CR2: 000000000207c930 CR3: 00000000018ce000 CR4: 00000000000007e0
[ 1088.311302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1088.311344] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1088.311386] Stack:
[ 1088.311400]  0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
[ 1088.311455]  000000000000057c 0000000000000052 ffff880beca94600 0000000100000000
[ 1088.311509]  ffffffffffffffd6 000000000000006c 0000002a0000053a ffff880beca94300
[ 1088.311563] Call Trace:
[ 1088.311580]  <IRQ> 
[ 1088.311596] 
[ 1088.311613]  [<ffffffff81532cad>] tcp_gso_segment+0x10d/0x3f0
[ 1088.311643]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
[ 1088.311681]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
[ 1088.311719]  [<ffffffff815522a4>] gre_gso_segment+0x134/0x370
[ 1088.311756]  [<ffffffff814d19f8>] ? dev_hard_start_xmit+0x318/0x570
[ 1088.311795]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
[ 1088.311836]  [<ffffffffa03cd107>] ? ipt_do_table+0x327/0x6d1 [ip_tables]
[ 1088.311883]  [<ffffffffa049d611>] ? __nf_conntrack_find_get+0x31/0x160 [nf_conntrack]
[ 1088.311931]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
[ 1088.311969]  [<ffffffff814d155d>] __skb_gso_segment+0x5d/0xb0
[ 1088.312005]  [<ffffffff814d186a>] dev_hard_start_xmit+0x18a/0x570
[ 1088.312044]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
[ 1088.312081]  [<ffffffff814efcfb>] ? eth_header+0x2b/0xd0
[ 1088.312116]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
[ 1088.312155]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
[ 1088.312193]  [<ffffffff8150f078>] ip_output+0x58/0x90
[ 1088.312225]  [<ffffffff8150e7e5>] ip_local_out+0x25/0x30
[ 1088.312259]  [<ffffffff81551afd>] iptunnel_xmit+0x18d/0x1d0
[ 1088.312299]  [<ffffffffa05443a7>] gre_tnl_send+0x1e7/0x390 [openvswitch]
[ 1088.312344]  [<ffffffffa0543ffd>] ovs_vport_send+0x1d/0x80 [openvswitch]
[ 1088.312387]  [<ffffffffa053a17a>] do_output+0x2a/0x50 [openvswitch]
[ 1088.312428]  [<ffffffffa053a653>] do_execute_actions+0x2e3/0xb60 [openvswitch]
[ 1088.312474]  [<ffffffffa053aefb>] ovs_execute_actions+0x2b/0x30 [openvswitch]
[ 1088.312521]  [<ffffffffa053d97f>] ovs_dp_process_received_packet+0x8f/0x120 [openvswitch]
[ 1088.314415]  [<ffffffffa03cd107>] ? ipt_do_table+0x327/0x6d1 [ip_tables]
[ 1088.316312]  [<ffffffffa0543f3a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[ 1088.318217]  [<ffffffffa05449dd>] internal_dev_xmit+0x1d/0x30 [openvswitch]
[ 1088.320109]  [<ffffffff814d19f8>] dev_hard_start_xmit+0x318/0x570
[ 1088.322014]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
[ 1088.323923]  [<ffffffff8150d970>] ? ip_fragment+0x870/0x870
[ 1088.325837]  [<ffffffff814efcfb>] ? eth_header+0x2b/0xd0
[ 1088.327744]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
[ 1088.329655]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
[ 1088.331551]  [<ffffffff8150f078>] ip_output+0x58/0x90
[ 1088.333424]  [<ffffffff8150afdb>] ip_forward_finish+0x8b/0x170
[ 1088.335299]  [<ffffffff8150b415>] ip_forward+0x355/0x400
[ 1088.337118]  [<ffffffff8150913d>] ip_rcv_finish+0x7d/0x350
[ 1088.338880]  [<ffffffff81509a04>] ip_rcv+0x234/0x380
[ 1088.340580]  [<ffffffff814cfc46>] __netif_receive_skb_core+0x676/0x870
[ 1088.342254]  [<ffffffff814cffed>] ? napi_gro_complete+0x7d/0x100
[ 1088.343859]  [<ffffffff814cfe58>] __netif_receive_skb+0x18/0x60
[ 1088.345405]  [<ffffffff814d0a1e>] process_backlog+0xae/0x180
[ 1088.346897]  [<ffffffff814d02aa>] net_rx_action+0x15a/0x250
[ 1088.348356]  [<ffffffff81067047>] __do_softirq+0xf7/0x290
[ 1088.349794]  [<ffffffff815f40dc>] call_softirq+0x1c/0x30
[ 1088.351214]  [<ffffffff81014d25>] do_softirq+0x55/0x90
[ 1088.352602]  [<ffffffff810673e5>] irq_exit+0x115/0x120
[ 1088.353971]  [<ffffffff815f49d8>] do_IRQ+0x58/0xf0
[ 1088.355323]  [<ffffffff815e9b2d>] common_interrupt+0x6d/0x6d
[ 1088.356679]  <EOI> 
[ 1088.356694] 
[ 1088.358034]  [<ffffffff810bc087>] ? clockevents_notify+0x127/0x140
[ 1088.359404]  [<ffffffff8148324f>] ? cpuidle_enter_state+0x4f/0xc0
[ 1088.360766]  [<ffffffff81483385>] cpuidle_idle_call+0xc5/0x200
[ 1088.362122]  [<ffffffff8101bcae>] arch_cpu_idle+0xe/0x30
[ 1088.363462]  [<ffffffff810b47b5>] cpu_startup_entry+0xf5/0x290
[ 1088.364793]  [<ffffffff815cff11>] start_secondary+0x265/0x27b
[ 1088.366125] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 40 b1 86 81 44 89 ce 31 c0 e8 ed 75 11 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 1a 0d 12 00 48 c7 c0 ea ff ff ff e9 c7 fd ff ff 
[ 1088.369119] RIP  [<ffffffff814c413a>] skb_segment+0x93a/0x960
[ 1088.370537]  RSP <ffff880bffce3478>

Environment

  • Red Hat Enterprise Linux OpenStack Platform 5.0 on RHEL7
  • VXLAN for tenant network

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In