System starts to show slow network performance and continuous kernel Oops with Red Hat Openstack 4.0
Issue
- After upgrade OpenStack from 3.0 to 4.0 and upgrade kernel to RHEL 6.5, the users reported exceptionally slow network speeds
- From the log it shows lots of Kernel Oops messages.
:WARNING: at net/core/dev.c:1907 skb_warn_bad_offload+0xc2/0xf0() (Tainted: G W --------------- )
:Hardware name: PowerEdge R620
:bonding: caps=(0xf153a5, 0x0) len=2962 data_len=2896 ip_summed=3
:Modules linked in: bridge xt_conntrack ipt_REDIRECT dlm configfs iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 veth ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables bonding 8021q garp stp llc ipv6 openvswitch vxlan iTCO_wdt iTCO_vendor_support microcode dcdbas power_meter sb_edac edac_core lpc_ich mfd_core shpchp igb i2c_algo_bit i2c_core sg ixgbe dca ptp pps_core mdio ext4 jbd2 mbcache sd_mod crc_t10dif ahci wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
:Pid: 0, comm: swapper Tainted: G W --------------- 2.6.32-431.3.1.el6.x86_64 #1
:Call Trace:
:<IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
:[<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
:[<ffffffff8145afe2>] ? skb_warn_bad_offload+0xc2/0xf0
:[<ffffffff8145bd4c>] ? skb_checksum_help+0x19c/0x1c0
:[<ffffffff81460495>] ? dev_hard_start_xmit+0x355/0x480
:[<ffffffff814607fd>] ? dev_queue_xmit+0x1bd/0x320
:[<ffffffffa01e921d>] ? netdev_send+0x4d/0xc0 [openvswitch]
:[<ffffffffa01e7fe2>] ? ovs_vport_send+0x22/0x90 [openvswitch]
:[<ffffffffa01e13fe>] ? do_output+0x2e/0x50 [openvswitch]
:[<ffffffffa01e1cb6>] ? do_execute_actions+0x896/0xbd0 [openvswitch]
:[<ffffffff81460364>] ? dev_hard_start_xmit+0x224/0x480
:[<ffffffff812920a2>] ? flex_array_get+0x12/0x80
:[<ffffffffa01e6b46>] ? find_bucket+0x66/0x70 [openvswitch]
:[<ffffffffa01e6d61>] ? ovs_flow_tbl_lookup+0x51/0xb0 [openvswitch]
:[<ffffffffa01e201b>] ? ovs_execute_actions+0x2b/0x30 [openvswitch]
:[<ffffffffa01e3a3c>] ? ovs_dp_process_received_packet+0xac/0x130 [openvswitch]
:[<ffffffffa01e807e>] ? ovs_vport_receive+0x2e/0x30 [openvswitch]
:[<ffffffffa01e9513>] ? ovs_netdev_frame_hook+0xb3/0x110 [openvswitch]
:[<ffffffff810a7159>] ? ktime_get+0x69/0xf0
:[<ffffffff8145b6ba>] ? __netif_receive_skb+0x60a/0x750
:[<ffffffff810ac905>] ? tick_dev_program_event+0x65/0xc0
:[<ffffffff8145b89a>] ? process_backlog+0x9a/0x100
:[<ffffffff81460b53>] ? net_rx_action+0x103/0x2f0
:[<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
:[<ffffffff810e6ed0>] ? handle_IRQ_event+0x60/0x170
:[<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
:[<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
:[<ffffffff8107a795>] ? irq_exit+0x85/0x90
:[<ffffffff81530ff5>] ? do_IRQ+0x75/0xf0
:[<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
:<EOI> [<ffffffff812e09ce>] ? intel_idle+0xde/0x170
:[<ffffffff812e09b1>] ? intel_idle+0xc1/0x170
:[<ffffffff81426707>] ? cpuidle_idle_call+0xa7/0x140
:[<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
:[<ffffffff81520e2c>] ? start_secondary+0x2ac/0x2ef
- Switching back to the old kernel 2.6.32-358.123.4.openstack.el6.x86_64 resolves the issue
Environment
- Red Hat OpenStack 4.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.