kernel BUG at net/core/skbuff.c:2979! - RHEL7OSP5 Neutron controller kernel panic due to GSO offloading.

Issue

• RHEL 7 OSP 5 Neutron controller crashes shortly after instances are started with the following trace present under /var/crash/ :

1057 [   63.461547] kernel BUG at net/core/skbuff.c:2979!
1058 [   63.461580] invalid opcode: 0000 [#1] SMP
1059 [   63.461598] Modules linked in: xt_nat xt_REDIRECT ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat veth openvswitch vxlan ip_tunnel gre sg ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_conntrack nf_conntrack xt_multiport iptable_filter ip_tabl     es ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support coretemp ipmi_devintf dcdbas kvm_intel kvm bnx2 i7core_edac crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ses aesni_intel lrw edac_core acpi_power_meter gf128mul glue_helper ablk_helper enclosure cryp     td serio_raw pcspkr shpchp lpc_ich ipmi_si ipmi_msghandler mfd_core mperf xfs libcrc32c sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm i2c_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
1060 [   63.461933] CPU: 19 PID: 0 Comm: swapper/19 Tainted: G          I --------------   3.10.0-123.6.3.el7.x86_64 #1
1061 [   63.461967] Hardware name: Dell Inc. PowerEdge R710/0MD99X, BIOS 2.2.10 11/09/2010
1062 [   63.461993] task: ffff88081964b8e0 ti: ffff880819658000 task.ti: ffff880819658000
1063 [   63.462019] RIP: 0010:[<ffffffff814c413a>]  [<ffffffff814c413a>] skb_segment+0x93a/0x960
1064 [   63.462052] RSP: 0018:ffff88041fd23478  EFLAGS: 00010206
1065 [   63.462071] RAX: 0000000000000000 RBX: ffff8800caa88400 RCX: ffff8800ca7626f0
1066 [   63.462095] RDX: 0000000000000050 RSI: ffff8800ca762600 RDI: ffff8800ca762a00
1067 [   63.462119] RBP: ffff88041fd23530 R08: 0000000000000042 R09: 0000000000000000
1068 [   63.462144] R10: ffff880036c7d500 R11: 00000000000005ea R12: ffff8800ca762af0
1069 [   63.462168] R13: 0000000000000000 R14: ffff880036c7c000 R15: 0000000000000050
1070 [   63.462192] FS:  0000000000000000(0000) GS:ffff88041fd20000(0000) knlGS:0000000000000000
1071 [   63.462220] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
1072 [   63.462239] CR2: 00007ff809eda820 CR3: 00000000018ce000 CR4: 00000000000007e0
1073 [   63.462264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
1074 [   63.462288] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
1075 [   63.462312] Stack:
1076 [   63.462320]  0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
1077 [   63.462353]  00000000000005ea 0000000000000052 ffff880036c7d500 0000000100000000
1078 [   63.462385]  ffffffffffffffd6 000000000000006c 0000002a000005a8 ffff8800caa88400
1079 [   63.462417] Call Trace:
1080 [   63.462427]  <IRQ>
1081 [   63.462437]
1082 [   63.462448]  [<ffffffff81532cad>] tcp_gso_segment+0x10d/0x3f0
1083 [   63.462466]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
1084 [   63.462489]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
1085 [   63.462511]  [<ffffffff815522a4>] gre_gso_segment+0x134/0x370
1086 [   63.462532]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
1087 [   63.462555]  [<ffffffff8106f7d6>] ? mod_timer_pending+0x76/0x1b0
1088 [   63.462582]  [<ffffffffa03afc00>] ? ____nf_conntrack_find+0x40/0x160 [nf_conntrack]
1089 [   63.462610]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
1090 [   63.462632]  [<ffffffff814d155d>] __skb_gso_segment+0x5d/0xb0
1091 [   63.462653]  [<ffffffff814d186a>] dev_hard_start_xmit+0x18a/0x570
1092 [   63.462678]  [<ffffffff814f0bee>] sch_direct_xmit+0xee/0x1c0
1093 [   63.462699]  [<ffffffff814d1e48>] dev_queue_xmit+0x1f8/0x4a0
1094 [   63.462721]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
1095 [   63.463535]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
1096 [   63.464352]  [<ffffffff8150f078>] ip_output+0x58/0x90
1097 [   63.465162]  [<ffffffff8150e7e5>] ip_local_out+0x25/0x30
1098 [   63.465970]  [<ffffffff81551afd>] iptunnel_xmit+0x18d/0x1d0
1099 [   63.466770]  [<ffffffffa03ed3a7>] gre_tnl_send+0x1e7/0x390 [openvswitch]
1100 [   63.467568]  [<ffffffffa03ecffd>] ovs_vport_send+0x1d/0x80 [openvswitch]
1101 [   63.468362]  [<ffffffffa03e317a>] do_output+0x2a/0x50 [openvswitch]
1102 [   63.469137]  [<ffffffffa03e3653>] do_execute_actions+0x2e3/0xb60 [openvswitch]
1103 [   63.469896]  [<ffffffffa03e3efb>] ovs_execute_actions+0x2b/0x30 [openvswitch]
1104 [   63.470634]  [<ffffffffa03e697f>] ovs_dp_process_received_packet+0x8f/0x120 [openvswitch]
1105 [   63.471359]  [<ffffffff814e1e06>] ? inet_proto_csum_replace4+0xa6/0xf0
1106 [   63.472064]  [<ffffffffa03ecf3a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
1107 [   63.472752]  [<ffffffffa03ed9dd>] internal_dev_xmit+0x1d/0x30 [openvswitch]
1108 [   63.473414]  [<ffffffff814d19f8>] dev_hard_start_xmit+0x318/0x570
1109 [   63.474061]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
1110 [   63.474690]  [<ffffffff8150d970>] ? ip_fragment+0x870/0x870
1111 [   63.475309]  [<ffffffff8150dc41>] ip_finish_output+0x2d1/0x3b0
1112 [   63.475918]  [<ffffffff8150f078>] ip_output+0x58/0x90
1113 [   63.476515]  [<ffffffff8150afdb>] ip_forward_finish+0x8b/0x170
1114 [   63.477107]  [<ffffffff8150b415>] ip_forward+0x355/0x400
1115 [   63.477700]  [<ffffffff8150913d>] ip_rcv_finish+0x7d/0x350
1116 [   63.478287]  [<ffffffff81509a04>] ip_rcv+0x234/0x380
1117 [   63.478870]  [<ffffffff814cfc46>] __netif_receive_skb_core+0x676/0x870
1118 [   63.479457]  [<ffffffff814cffed>] ? napi_gro_complete+0x7d/0x100
1119 [   63.480038]  [<ffffffff814cfe58>] __netif_receive_skb+0x18/0x60
1120 [   63.480610]  [<ffffffff814d0a1e>] process_backlog+0xae/0x180
1121 [   63.481176]  [<ffffffff814d02aa>] net_rx_action+0x15a/0x250
1122 [   63.481743]  [<ffffffff81067047>] __do_softirq+0xf7/0x290
1123 [   63.482315]  [<ffffffff815f40dc>] call_softirq+0x1c/0x30
1124 [   63.482875]  [<ffffffff81014d25>] do_softirq+0x55/0x90
1125 [   63.483423]  [<ffffffff810673e5>] irq_exit+0x115/0x120
1126 [   63.483968]  [<ffffffff815f49d8>] do_IRQ+0x58/0xf0
1127 [   63.484506]  [<ffffffff815e9b2d>] common_interrupt+0x6d/0x6d
1128 [   63.485036]  <EOI>
1129 [   63.485045]
1130 [   63.485575]  [<ffffffff8148324f>] ? cpuidle_enter_state+0x4f/0xc0
1131 [   63.486115]  [<ffffffff81483385>] cpuidle_idle_call+0xc5/0x200
1132 [   63.486659]  [<ffffffff8101bcae>] arch_cpu_idle+0xe/0x30
1133 [   63.487202]  [<ffffffff810b47b5>] cpu_startup_entry+0xf5/0x290
1134 [   63.487745]  [<ffffffff815cff11>] start_secondary+0x265/0x27b
1135 [   63.488286] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 40 b1 86 81 44 89 ce 31 c0 e8 ed 75 11 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 1a 0d 12 00 48 c7 c0 ea ff ff ff e9 c7 fd ff ff
1136 [   63.489522] RIP  [<ffffffff814c413a>] skb_segment+0x93a/0x960
1137 [   63.490099]  RSP <ffff88041fd23478>