The kernel is getting unresponsive/hung where lots of tasks are performing memory compaction

Solution Unverified - Updated -

Issue

  • The kernel is getting unresponsive/hung where lots of tasks are performing memory compaction.

  • Lots of kworkers are waiting for kthreadd to create workers and getting blocked in TASK_UNINTERRUPTIBLE sleep state for quite a long time with backtraces like this:

PID: 23151  TASK: ffff9a2d3db12000  CPU: 16  COMMAND: "kworker/16:3"
 #0 [ffff9a5813fc7b68] __schedule at ffffffffb3581dea
 #1 [ffff9a5813fc7bf8] schedule at ffffffffb3582299
 #2 [ffff9a5813fc7c08] schedule_timeout at ffffffffb357fda1
 #3 [ffff9a5813fc7cb8] wait_for_completion at ffffffffb358264d
 #4 [ffff9a5813fc7d18] kthread_create_on_node at ffffffffb2ec61fa
 #5 [ffff9a5813fc7dd0] create_worker at ffffffffb2ebef8b
 #6 [ffff9a5813fc7e20] manage_workers at ffffffffb2ebf216
 #7 [ffff9a5813fc7e68] worker_thread at ffffffffb2ebf743
 #8 [ffff9a5813fc7ec8] kthread at ffffffffb2ec63a1
  • At that time kthreadd is trying to create a worker and thus performing direct memory compaction with a backtrace like this:
PID: 2      TASK: ffff9a4cd3981070  CPU: 14  COMMAND: "kthreadd"
 #0 [ffff9a2d537d3878] __schedule at ffffffffb3581dea
 #1 [ffff9a2d537d3908] __cond_resched at ffffffffb2ed7c36
 #2 [ffff9a2d537d3920] _cond_resched at ffffffffb358253a
 #3 [ffff9a2d537d3930] compact_checklock_irqsave at ffffffffb2fe45ad
 #4 [ffff9a2d537d3960] isolate_migratepages_range at ffffffffb2fe545a
 #5 [ffff9a2d537d3a20] compact_zone at ffffffffb2fe5b03
 #6 [ffff9a2d537d3a78] compact_zone_order at ffffffffb2fe5d2d
 #7 [ffff9a2d537d3b10] try_to_compact_pages at ffffffffb2fe60d1
 #8 [ffff9a2d537d3b70] __alloc_pages_direct_compact at ffffffffb357728e
 #9 [ffff9a2d537d3bd0] __alloc_pages_slowpath at ffffffffb357775e
#10 [ffff9a2d537d3cc0] __alloc_pages_nodemask at ffffffffb2fc8796
#11 [ffff9a2d537d3d68] copy_process at ffffffffb2e9914d
#12 [ffff9a2d537d3df8] do_fork at ffffffffb2e9ab71
#13 [ffff9a2d537d3e70] kernel_thread at ffffffffb2e9ae36
#14 [ffff9a2d537d3e80] kthreadd at ffffffffb2ec6d81

Environment

  • Red Hat Enterprise Linux 7.7 (kernel-3.10.0-1062.37.1.el7)
  • Transparent Hugepages disabled
  • Hugepages enabled

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content