Soft lockup on RHEL 6.4 with backtrace in mem_cgroup_zone_nr_pages
Issue
- kswapd0/khugepaged is causing 100% CPU with RHEL6.4 kernel
- Machine crashes with this error:
<0>BUG: soft lockup - CPU#5 stuck for 67s! [vhost-14260:14283]$
- Prints a backtrace containing lines similar to this:
<4>Call Trace:$
. . .
<4> [<ffffffff81133900>] ? shrink_mem_cgroup_zone+0xd0/0x610$
<4> [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280$
<4> [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0$
<4> [<ffffffff81134aa9>] ? zone_reclaim+0x349/0x400$
<4> [<ffffffff8112a62c>] ? get_page_from_freelist+0x69c/0x830$
<4> [<ffffffff81277349>] ? cpumask_next_and+0x29/0x50$
<4> [<ffffffff81065905>] ? enqueue_entity+0x125/0x410$
<4> [<ffffffff8112bbc3>] ? __alloc_pages_nodemask+0x113/0x8d0$
<4> [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90$
<4> [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0$
<4> [<ffffffff81063322>] ? default_wake_function+0x12/0x20$
<4> [<ffffffff81051419>] ? __wake_up_common+0x59/0x90$
<4> [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170$
<4> [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320$
<4> [<ffffffff81167372>] ? cache_alloc_refill+0x202/0x240$
<4> [<ffffffff8116876a>] ? kmem_cache_alloc_node_trace+0x1ea/0x200$
<4> [<ffffffff8116882d>] ? __kmalloc_node+0x4d/0x60$
<4> [<ffffffff8143d8cd>] ? __alloc_skb+0x6d/0x190$
. . .
Environment
- Red Hat Enterprise Linux 6.4
- 2.6.32-358.2.1.el6
- System running a number of KVM virtual machines
- KVM machines are using vhost networking
- System under memory pressure (several gigabytes of swap in use)
- System under network load
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
