Kernel panic of network node when guests access network

Solution Verified - Updated -

Issue

Network Node panics with the following back trace when load is applied on the Tenant networks

[ 1088.309799] ------------[ cut here ]------------
[ 1088.309893] kernel BUG at net/core/skbuff.c:2979!
[ 1088.309923] invalid opcode: 0000 [#1] SMP 
[ 1088.309954] Modules linked in: ip6t_rpfilter ip6t_REJECT ebtable_nat ebtable_broute ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ip6table_filter ip6_tables xt_nat xt_conntrack xt_REDIRECT veth 8021q garp mrp ipt_REJECT bridge openvswitch stp vxlan llc ip_tunnel gre libcrc32c iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables sg iTCO_wdt ipmi_devintf iTCO_vendor_support dcdbas coretemp kvm_intel kvm crc32c_intel pcspkr lpc_ich mfd_core ipmi_si ipmi_msghandler bnx2 i7core_edac edac_core acpi_power_meter shpchp nfsd mperf auth_rpcgss nfs_acl lockd sunrpc ext4 mbcache jbd2 dm_service_time sr_mod cdrom sd_mod crc_t10dif crct10dif_common
[ 1088.310573]  usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm qla2xxx i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_multipath dm_mod [last unloaded: iptable_raw]
[ 1088.310734] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G          I --------------   3.10.0-123.6.3.el7.x86_64 #1
[ 1088.310792] Hardware name: Dell Inc. PowerEdge M610/0N582M, BIOS 1.3.6 10/30/2009
[ 1088.310837] task: ffff8817ece90b60 ti: ffff8817ece8e000 task.ti: ffff8817ece8e000
[ 1088.310881] RIP: 0010:[<ffffffff814c413a>]  [<ffffffff814c413a>] skb_segment+0x93a/0x960
[ 1088.310935] RSP: 0018:ffff880bffce3478  EFLAGS: 00010202
[ 1088.310968] RAX: 0000000000000000 RBX: ffff880beca94300 RCX: ffff880be8d3a0f0
[ 1088.311010] RDX: 0000000000000050 RSI: ffff880be8d3a000 RDI: ffff880be8d3b200
[ 1088.311052] RBP: ffff880bffce3530 R08: 0000000000000042 R09: 0000000000000000
[ 1088.311094] R10: ffff880beca94600 R11: 000000000000057c R12: ffff880be8d3b2f0
[ 1088.311136] R13: 0000000000000000 R14: ffff880beca94500 R15: 0000000000000050
[ 1088.311178] FS:  0000000000000000(0000) GS:ffff880bffce0000(0000) knlGS:0000000000000000
[ 1088.311226] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1088.311260] CR2: 000000000207c930 CR3: 00000000018ce000 CR4: 00000000000007e0
[ 1088.311302] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1088.311344] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1088.311386] Stack:
[ 1088.311400]  0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
[ 1088.311455]  000000000000057c 0000000000000052 ffff880beca94600 0000000100000000
[ 1088.311509]  ffffffffffffffd6 000000000000006c 0000002a0000053a ffff880beca94300
[ 1088.311563] Call Trace:
[ 1088.311580]  <IRQ> 
[ 1088.311596] 
[ 1088.311613]  [<ffffffff81532cad>] tcp_gso_segment+0x10d/0x3f0
[ 1088.311643]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
[ 1088.311681]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
[ 1088.311719]  [<ffffffff815522a4>] gre_gso_segment+0x134/0x370
[ 1088.311756]  [<ffffffff814d19f8>] ? dev_hard_start_xmit+0x318/0x570
[ 1088.311795]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
[ 1088.311836]  [<ffffffffa03cd107>] ? ipt_do_table+0x327/0x6d1 [ip_tables]
[ 1088.311883]  [<ffffffffa049d611>] ? __nf_conntrack_find_get+0x31/0x160 [nf_conntrack]
[ 1088.311931]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
[ 1088.311969]  [<ffffffff814d155d>] __skb_gso_segment+0x5d/0xb0
[ 1088.312005]  [<ffffffff814d186a>] dev_hard_start_xmit+0x18a/0x570
[ 1088.312044]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
[ 1088.312081]  [<ffffffff814efcfb>] ? eth_header+0x2b/0xd0
[ 1088.312116]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
[ 1088.312155]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
[ 1088.312193]  [<ffffffff8150f078>] ip_output+0x58/0x90
[ 1088.312225]  [<ffffffff8150e7e5>] ip_local_out+0x25/0x30
[ 1088.312259]  [<ffffffff81551afd>] iptunnel_xmit+0x18d/0x1d0
[ 1088.312299]  [<ffffffffa05443a7>] gre_tnl_send+0x1e7/0x390 [openvswitch]
[ 1088.312344]  [<ffffffffa0543ffd>] ovs_vport_send+0x1d/0x80 [openvswitch]
[ 1088.312387]  [<ffffffffa053a17a>] do_output+0x2a/0x50 [openvswitch]
[ 1088.312428]  [<ffffffffa053a653>] do_execute_actions+0x2e3/0xb60 [openvswitch]
[ 1088.312474]  [<ffffffffa053aefb>] ovs_execute_actions+0x2b/0x30 [openvswitch]
[ 1088.312521]  [<ffffffffa053d97f>] ovs_dp_process_received_packet+0x8f/0x120 [openvswitch]
[ 1088.314415]  [<ffffffffa03cd107>] ? ipt_do_table+0x327/0x6d1 [ip_tables]
[ 1088.316312]  [<ffffffffa0543f3a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
[ 1088.318217]  [<ffffffffa05449dd>] internal_dev_xmit+0x1d/0x30 [openvswitch]
[ 1088.320109]  [<ffffffff814d19f8>] dev_hard_start_xmit+0x318/0x570
[ 1088.322014]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
[ 1088.323923]  [<ffffffff8150d970>] ? ip_fragment+0x870/0x870
[ 1088.325837]  [<ffffffff814efcfb>] ? eth_header+0x2b/0xd0
[ 1088.327744]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
[ 1088.329655]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
[ 1088.331551]  [<ffffffff8150f078>] ip_output+0x58/0x90
[ 1088.333424]  [<ffffffff8150afdb>] ip_forward_finish+0x8b/0x170
[ 1088.335299]  [<ffffffff8150b415>] ip_forward+0x355/0x400
[ 1088.337118]  [<ffffffff8150913d>] ip_rcv_finish+0x7d/0x350
[ 1088.338880]  [<ffffffff81509a04>] ip_rcv+0x234/0x380
[ 1088.340580]  [<ffffffff814cfc46>] __netif_receive_skb_core+0x676/0x870
[ 1088.342254]  [<ffffffff814cffed>] ? napi_gro_complete+0x7d/0x100
[ 1088.343859]  [<ffffffff814cfe58>] __netif_receive_skb+0x18/0x60
[ 1088.345405]  [<ffffffff814d0a1e>] process_backlog+0xae/0x180
[ 1088.346897]  [<ffffffff814d02aa>] net_rx_action+0x15a/0x250
[ 1088.348356]  [<ffffffff81067047>] __do_softirq+0xf7/0x290
[ 1088.349794]  [<ffffffff815f40dc>] call_softirq+0x1c/0x30
[ 1088.351214]  [<ffffffff81014d25>] do_softirq+0x55/0x90
[ 1088.352602]  [<ffffffff810673e5>] irq_exit+0x115/0x120
[ 1088.353971]  [<ffffffff815f49d8>] do_IRQ+0x58/0xf0
[ 1088.355323]  [<ffffffff815e9b2d>] common_interrupt+0x6d/0x6d
[ 1088.356679]  <EOI> 
[ 1088.356694] 
[ 1088.358034]  [<ffffffff810bc087>] ? clockevents_notify+0x127/0x140
[ 1088.359404]  [<ffffffff8148324f>] ? cpuidle_enter_state+0x4f/0xc0
[ 1088.360766]  [<ffffffff81483385>] cpuidle_idle_call+0xc5/0x200
[ 1088.362122]  [<ffffffff8101bcae>] arch_cpu_idle+0xe/0x30
[ 1088.363462]  [<ffffffff810b47b5>] cpu_startup_entry+0xf5/0x290
[ 1088.364793]  [<ffffffff815cff11>] start_secondary+0x265/0x27b
[ 1088.366125] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 40 b1 86 81 44 89 ce 31 c0 e8 ed 75 11 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 1a 0d 12 00 48 c7 c0 ea ff ff ff e9 c7 fd ff ff 
[ 1088.369119] RIP  [<ffffffff814c413a>] skb_segment+0x93a/0x960
[ 1088.370537]  RSP <ffff880bffce3478>

Environment

  • Red Hat Enterprise Linux OpenStack Platform 5.0 on RHEL7
  • VXLAN for tenant network

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content