System starts to show slow network performance and continuous kernel Oops with Red Hat Openstack 4.0

Solution Verified - Updated -

Issue

  • After upgrade OpenStack from 3.0 to 4.0 and upgrade kernel to RHEL 6.5, the users reported exceptionally slow network speeds
  • From the log it shows lots of Kernel Oops messages.
:WARNING: at net/core/dev.c:1907 skb_warn_bad_offload+0xc2/0xf0() (Tainted: G        W  ---------------   )
:Hardware name: PowerEdge R620
:bonding: caps=(0xf153a5, 0x0) len=2962 data_len=2896 ip_summed=3
:Modules linked in: bridge xt_conntrack ipt_REDIRECT dlm configfs iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 veth ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables bonding 8021q garp stp llc ipv6 openvswitch vxlan iTCO_wdt iTCO_vendor_support microcode dcdbas power_meter sb_edac edac_core lpc_ich mfd_core shpchp igb i2c_algo_bit i2c_core sg ixgbe dca ptp pps_core mdio ext4 jbd2 mbcache sd_mod crc_t10dif ahci wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
:Pid: 0, comm: swapper Tainted: G        W  ---------------    2.6.32-431.3.1.el6.x86_64 #1
:Call Trace:
:<IRQ>  [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
:[<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
:[<ffffffff8145afe2>] ? skb_warn_bad_offload+0xc2/0xf0
:[<ffffffff8145bd4c>] ? skb_checksum_help+0x19c/0x1c0
:[<ffffffff81460495>] ? dev_hard_start_xmit+0x355/0x480
:[<ffffffff814607fd>] ? dev_queue_xmit+0x1bd/0x320
:[<ffffffffa01e921d>] ? netdev_send+0x4d/0xc0 [openvswitch]
:[<ffffffffa01e7fe2>] ? ovs_vport_send+0x22/0x90 [openvswitch]
:[<ffffffffa01e13fe>] ? do_output+0x2e/0x50 [openvswitch]
:[<ffffffffa01e1cb6>] ? do_execute_actions+0x896/0xbd0 [openvswitch]
:[<ffffffff81460364>] ? dev_hard_start_xmit+0x224/0x480
:[<ffffffff812920a2>] ? flex_array_get+0x12/0x80
:[<ffffffffa01e6b46>] ? find_bucket+0x66/0x70 [openvswitch]
:[<ffffffffa01e6d61>] ? ovs_flow_tbl_lookup+0x51/0xb0 [openvswitch]
:[<ffffffffa01e201b>] ? ovs_execute_actions+0x2b/0x30 [openvswitch]
:[<ffffffffa01e3a3c>] ? ovs_dp_process_received_packet+0xac/0x130 [openvswitch]
:[<ffffffffa01e807e>] ? ovs_vport_receive+0x2e/0x30 [openvswitch]
:[<ffffffffa01e9513>] ? ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
:[<ffffffff810a7159>] ? ktime_get+0x69/0xf0
:[<ffffffff8145b6ba>] ? __netif_receive_skb+0x60a/0x750
:[<ffffffff810ac905>] ? tick_dev_program_event+0x65/0xc0
:[<ffffffff8145b89a>] ? process_backlog+0x9a/0x100
:[<ffffffff81460b53>] ? net_rx_action+0x103/0x2f0
:[<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
:[<ffffffff810e6ed0>] ? handle_IRQ_event+0x60/0x170
:[<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
:[<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
:[<ffffffff8107a795>] ? irq_exit+0x85/0x90
:[<ffffffff81530ff5>] ? do_IRQ+0x75/0xf0
:[<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
:<EOI>  [<ffffffff812e09ce>] ? intel_idle+0xde/0x170
:[<ffffffff812e09b1>] ? intel_idle+0xc1/0x170
:[<ffffffff81426707>] ? cpuidle_idle_call+0xa7/0x140
:[<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
:[<ffffffff81520e2c>] ? start_secondary+0x2ac/0x2ef
  • Switching back to the old kernel 2.6.32-358.123.4.openstack.el6.x86_64 resolves the issue

Environment

  • Red Hat OpenStack 4.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content