kernel BUG at net/core/skbuff.c:2979! - RHEL7OSP5 Neutron controller kernel panic due to GSO offloading.

Solution Verified - Updated -

Issue

  • RHEL 7 OSP 5 Neutron controller crashes shortly after instances are started with the following trace present under /var/crash/ :
1057 [   63.461547] kernel BUG at net/core/skbuff.c:2979!
1058 [   63.461580] invalid opcode: 0000 [#1] SMP
1059 [   63.461598] Modules linked in: xt_nat xt_REDIRECT ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat veth openvswitch vxlan ip_tunnel gre sg ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_conntrack nf_conntrack xt_multiport iptable_filter ip_tabl     es ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support coretemp ipmi_devintf dcdbas kvm_intel kvm bnx2 i7core_edac crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ses aesni_intel lrw edac_core acpi_power_meter gf128mul glue_helper ablk_helper enclosure cryp     td serio_raw pcspkr shpchp lpc_ich ipmi_si ipmi_msghandler mfd_core mperf xfs libcrc32c sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm i2c_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
1060 [   63.461933] CPU: 19 PID: 0 Comm: swapper/19 Tainted: G          I --------------   3.10.0-123.6.3.el7.x86_64 #1
1061 [   63.461967] Hardware name: Dell Inc. PowerEdge R710/0MD99X, BIOS 2.2.10 11/09/2010
1062 [   63.461993] task: ffff88081964b8e0 ti: ffff880819658000 task.ti: ffff880819658000
1063 [   63.462019] RIP: 0010:[<ffffffff814c413a>]  [<ffffffff814c413a>] skb_segment+0x93a/0x960
1064 [   63.462052] RSP: 0018:ffff88041fd23478  EFLAGS: 00010206
1065 [   63.462071] RAX: 0000000000000000 RBX: ffff8800caa88400 RCX: ffff8800ca7626f0
1066 [   63.462095] RDX: 0000000000000050 RSI: ffff8800ca762600 RDI: ffff8800ca762a00
1067 [   63.462119] RBP: ffff88041fd23530 R08: 0000000000000042 R09: 0000000000000000
1068 [   63.462144] R10: ffff880036c7d500 R11: 00000000000005ea R12: ffff8800ca762af0
1069 [   63.462168] R13: 0000000000000000 R14: ffff880036c7c000 R15: 0000000000000050
1070 [   63.462192] FS:  0000000000000000(0000) GS:ffff88041fd20000(0000) knlGS:0000000000000000
1071 [   63.462220] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
1072 [   63.462239] CR2: 00007ff809eda820 CR3: 00000000018ce000 CR4: 00000000000007e0
1073 [   63.462264] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
1074 [   63.462288] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
1075 [   63.462312] Stack:
1076 [   63.462320]  0000000000000000 0000000000000000 0000000000000042 ffffffffffffffbe
1077 [   63.462353]  00000000000005ea 0000000000000052 ffff880036c7d500 0000000100000000
1078 [   63.462385]  ffffffffffffffd6 000000000000006c 0000002a000005a8 ffff8800caa88400
1079 [   63.462417] Call Trace:
1080 [   63.462427]  <IRQ>
1081 [   63.462437]
1082 [   63.462448]  [<ffffffff81532cad>] tcp_gso_segment+0x10d/0x3f0
1083 [   63.462466]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
1084 [   63.462489]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
1085 [   63.462511]  [<ffffffff815522a4>] gre_gso_segment+0x134/0x370
1086 [   63.462532]  [<ffffffff81542f62>] inet_gso_segment+0x122/0x340
1087 [   63.462555]  [<ffffffff8106f7d6>] ? mod_timer_pending+0x76/0x1b0
1088 [   63.462582]  [<ffffffffa03afc00>] ? ____nf_conntrack_find+0x40/0x160 [nf_conntrack]
1089 [   63.462610]  [<ffffffff814d140c>] skb_mac_gso_segment+0xac/0x1a0
1090 [   63.462632]  [<ffffffff814d155d>] __skb_gso_segment+0x5d/0xb0
1091 [   63.462653]  [<ffffffff814d186a>] dev_hard_start_xmit+0x18a/0x570
1092 [   63.462678]  [<ffffffff814f0bee>] sch_direct_xmit+0xee/0x1c0
1093 [   63.462699]  [<ffffffff814d1e48>] dev_queue_xmit+0x1f8/0x4a0
1094 [   63.462721]  [<ffffffff814d982b>] neigh_resolve_output+0x11b/0x220
1095 [   63.463535]  [<ffffffff8150db40>] ip_finish_output+0x1d0/0x3b0
1096 [   63.464352]  [<ffffffff8150f078>] ip_output+0x58/0x90
1097 [   63.465162]  [<ffffffff8150e7e5>] ip_local_out+0x25/0x30
1098 [   63.465970]  [<ffffffff81551afd>] iptunnel_xmit+0x18d/0x1d0
1099 [   63.466770]  [<ffffffffa03ed3a7>] gre_tnl_send+0x1e7/0x390 [openvswitch]
1100 [   63.467568]  [<ffffffffa03ecffd>] ovs_vport_send+0x1d/0x80 [openvswitch]
1101 [   63.468362]  [<ffffffffa03e317a>] do_output+0x2a/0x50 [openvswitch]
1102 [   63.469137]  [<ffffffffa03e3653>] do_execute_actions+0x2e3/0xb60 [openvswitch]
1103 [   63.469896]  [<ffffffffa03e3efb>] ovs_execute_actions+0x2b/0x30 [openvswitch]
1104 [   63.470634]  [<ffffffffa03e697f>] ovs_dp_process_received_packet+0x8f/0x120 [openvswitch]
1105 [   63.471359]  [<ffffffff814e1e06>] ? inet_proto_csum_replace4+0xa6/0xf0
1106 [   63.472064]  [<ffffffffa03ecf3a>] ovs_vport_receive+0x2a/0x30 [openvswitch]
1107 [   63.472752]  [<ffffffffa03ed9dd>] internal_dev_xmit+0x1d/0x30 [openvswitch]
1108 [   63.473414]  [<ffffffff814d19f8>] dev_hard_start_xmit+0x318/0x570
1109 [   63.474061]  [<ffffffff814d1f80>] dev_queue_xmit+0x330/0x4a0
1110 [   63.474690]  [<ffffffff8150d970>] ? ip_fragment+0x870/0x870
1111 [   63.475309]  [<ffffffff8150dc41>] ip_finish_output+0x2d1/0x3b0
1112 [   63.475918]  [<ffffffff8150f078>] ip_output+0x58/0x90
1113 [   63.476515]  [<ffffffff8150afdb>] ip_forward_finish+0x8b/0x170
1114 [   63.477107]  [<ffffffff8150b415>] ip_forward+0x355/0x400
1115 [   63.477700]  [<ffffffff8150913d>] ip_rcv_finish+0x7d/0x350
1116 [   63.478287]  [<ffffffff81509a04>] ip_rcv+0x234/0x380
1117 [   63.478870]  [<ffffffff814cfc46>] __netif_receive_skb_core+0x676/0x870
1118 [   63.479457]  [<ffffffff814cffed>] ? napi_gro_complete+0x7d/0x100
1119 [   63.480038]  [<ffffffff814cfe58>] __netif_receive_skb+0x18/0x60
1120 [   63.480610]  [<ffffffff814d0a1e>] process_backlog+0xae/0x180
1121 [   63.481176]  [<ffffffff814d02aa>] net_rx_action+0x15a/0x250
1122 [   63.481743]  [<ffffffff81067047>] __do_softirq+0xf7/0x290
1123 [   63.482315]  [<ffffffff815f40dc>] call_softirq+0x1c/0x30
1124 [   63.482875]  [<ffffffff81014d25>] do_softirq+0x55/0x90
1125 [   63.483423]  [<ffffffff810673e5>] irq_exit+0x115/0x120
1126 [   63.483968]  [<ffffffff815f49d8>] do_IRQ+0x58/0xf0
1127 [   63.484506]  [<ffffffff815e9b2d>] common_interrupt+0x6d/0x6d
1128 [   63.485036]  <EOI>
1129 [   63.485045]
1130 [   63.485575]  [<ffffffff8148324f>] ? cpuidle_enter_state+0x4f/0xc0
1131 [   63.486115]  [<ffffffff81483385>] cpuidle_idle_call+0xc5/0x200
1132 [   63.486659]  [<ffffffff8101bcae>] arch_cpu_idle+0xe/0x30
1133 [   63.487202]  [<ffffffff810b47b5>] cpu_startup_entry+0xf5/0x290
1134 [   63.487745]  [<ffffffff815cff11>] start_secondary+0x265/0x27b
1135 [   63.488286] Code: 0f 84 0c ff ff ff 8b 54 24 50 48 c7 c7 40 b1 86 81 44 89 ce 31 c0 e8 ed 75 11 00 48 8b 7c 24 58 e9 ff fe ff ff 0f 1f 40 00 0f 0b <0f> 0b 0f 0b e8 1a 0d 12 00 48 c7 c0 ea ff ff ff e9 c7 fd ff ff
1136 [   63.489522] RIP  [<ffffffff814c413a>] skb_segment+0x93a/0x960
1137 [   63.490099]  RSP <ffff88041fd23478>

Environment

  • Red Hat Enterprise Linux 7 Openstack Platform 5.
  • Neutron controller node with GSO enabled interfaces and GRE tunnels between nodes.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content