Soft lockup on RHEL 6.4 with backtrace in mem_cgroup_zone_nr_pages
Issue
- kswapd0/khugepaged is causing 100% CPU with RHEL6.4 kernel
- Machine crashes with this error:
<0>BUG: soft lockup - CPU#5 stuck for 67s! [vhost-14260:14283]$
- Prints a backtrace containing lines similar to this:
<4>Call Trace:$
. . .
<4> [<ffffffff81133900>] ? shrink_mem_cgroup_zone+0xd0/0x610$
<4> [<ffffffff8117257d>] ? mem_cgroup_iter+0xfd/0x280$
<4> [<ffffffff81133ea3>] ? shrink_zone+0x63/0xb0$
<4> [<ffffffff81134aa9>] ? zone_reclaim+0x349/0x400$
<4> [<ffffffff8112a62c>] ? get_page_from_freelist+0x69c/0x830$
<4> [<ffffffff81277349>] ? cpumask_next_and+0x29/0x50$
<4> [<ffffffff81065905>] ? enqueue_entity+0x125/0x410$
<4> [<ffffffff8112bbc3>] ? __alloc_pages_nodemask+0x113/0x8d0$
<4> [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90$
<4> [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0$
<4> [<ffffffff81063322>] ? default_wake_function+0x12/0x20$
<4> [<ffffffff81051419>] ? __wake_up_common+0x59/0x90$
<4> [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170$
<4> [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320$
<4> [<ffffffff81167372>] ? cache_alloc_refill+0x202/0x240$
<4> [<ffffffff8116876a>] ? kmem_cache_alloc_node_trace+0x1ea/0x200$
<4> [<ffffffff8116882d>] ? __kmalloc_node+0x4d/0x60$
<4> [<ffffffff8143d8cd>] ? __alloc_skb+0x6d/0x190$
. . .
Environment
- Red Hat Enterprise Linux 6.4
- 2.6.32-358.2.1.el6
- System running a number of KVM virtual machines
- KVM machines are using vhost networking
- System under memory pressure (several gigabytes of swap in use)
- System under network load
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.