Soft lockup on RHEL 6.4 with backtrace in mem_cgroup_zone_nr_pages

Solution Unverified - Updated -

Issue

  • kswapd0/khugepaged is causing 100% CPU with RHEL6.4 kernel
  • Machine crashes with this error:
<0>BUG: soft lockup - CPU#5 stuck for 67s! [vhost-14260:14283]$
  • Prints a backtrace containing lines similar to this:
<4>Call Trace:$
. . .
<4> [<ffffffff81133900>] ? shrink_mem_cgroup_zone+0xd0/0x610$
<4> [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280$
<4> [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0$
<4> [<ffffffff81134aa9>] ? zone_reclaim+0x349/0x400$
<4> [<ffffffff8112a62c>] ? get_page_from_freelist+0x69c/0x830$
<4> [<ffffffff81277349>] ? cpumask_next_and+0x29/0x50$
<4> [<ffffffff81065905>] ? enqueue_entity+0x125/0x410$
<4> [<ffffffff8112bbc3>] ? __alloc_pages_nodemask+0x113/0x8d0$
<4> [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90$
<4> [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0$
<4> [<ffffffff81063322>] ? default_wake_function+0x12/0x20$
<4> [<ffffffff81051419>] ? __wake_up_common+0x59/0x90$
<4> [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170$
<4> [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320$
<4> [<ffffffff81167372>] ? cache_alloc_refill+0x202/0x240$
<4> [<ffffffff8116876a>] ? kmem_cache_alloc_node_trace+0x1ea/0x200$
<4> [<ffffffff8116882d>] ? __kmalloc_node+0x4d/0x60$
<4> [<ffffffff8143d8cd>] ? __alloc_skb+0x6d/0x190$
. . .

Environment

  • Red Hat Enterprise Linux 6.4
    • 2.6.32-358.2.1.el6
  • System running a number of KVM virtual machines
    • KVM machines are using vhost networking
  • System under memory pressure (several gigabytes of swap in use)
  • System under network load

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content