Soft lockup on RHEL 6.4 with backtrace in mem_cgroup_zone_nr_pages

Solution Unverified - Updated -

Issue

  • kswapd0/khugepaged is causing 100% CPU with RHEL6.4 kernel
  • Machine crashes with this error:
<0>BUG: soft lockup - CPU#5 stuck for 67s! [vhost-14260:14283]$
  • Prints a backtrace containing lines similar to this:
<4>Call Trace:$
. . .
<4> [<ffffffff81133900>] ? shrink_mem_cgroup_zone+0xd0/0x610$
<4> [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280$
<4> [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0$
<4> [<ffffffff81134aa9>] ? zone_reclaim+0x349/0x400$
<4> [<ffffffff8112a62c>] ? get_page_from_freelist+0x69c/0x830$
<4> [<ffffffff81277349>] ? cpumask_next_and+0x29/0x50$
<4> [<ffffffff81065905>] ? enqueue_entity+0x125/0x410$
<4> [<ffffffff8112bbc3>] ? __alloc_pages_nodemask+0x113/0x8d0$
<4> [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90$
<4> [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0$
<4> [<ffffffff81063322>] ? default_wake_function+0x12/0x20$
<4> [<ffffffff81051419>] ? __wake_up_common+0x59/0x90$
<4> [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170$
<4> [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320$
<4> [<ffffffff81167372>] ? cache_alloc_refill+0x202/0x240$
<4> [<ffffffff8116876a>] ? kmem_cache_alloc_node_trace+0x1ea/0x200$
<4> [<ffffffff8116882d>] ? __kmalloc_node+0x4d/0x60$
<4> [<ffffffff8143d8cd>] ? __alloc_skb+0x6d/0x190$
. . .

Environment

  • Red Hat Enterprise Linux 6.4
    • 2.6.32-358.2.1.el6
  • System running a number of KVM virtual machines
    • KVM machines are using vhost networking
  • System under memory pressure (several gigabytes of swap in use)
  • System under network load

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.