Why kswapd is getting stuck in xfs_reclaim_inodes_ag ?

Issue

• kswapd processes getting blocked for more than 120 seconds

  kernel: INFO: task kswapd0:229 blocked for more than 120 seconds.
  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  kernel: kswapd0         D ffff8ebc1de49040     0   229      2 0x00000080
  kernel: Call Trace:
  kernel: [<ffffffff8d368b69>] schedule_preempt_disabled+0x29/0x70
  kernel: [<ffffffff8d366ab7>] __mutex_lock_slowpath+0xc7/0x1d0
  kernel: [<ffffffff8d365e9f>] mutex_lock+0x1f/0x2f
  kernel: [<ffffffffc06aeccc>] xfs_reclaim_inodes_ag+0x2dc/0x390 [xfs]
  kernel: [<ffffffff8cd53cdd>] ? call_rcu_sched+0x1d/0x20
  kernel: [<ffffffff8ce5998f>] ? d_free+0x4f/0x70
  kernel: [<ffffffff8ce59faa>] ? __dentry_kill+0x16a/0x180
  kernel: [<ffffffff8ce5a1f8>] ? shrink_dentry_list+0x58/0x230
  kernel: [<ffffffffc06afe33>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
  kernel: [<ffffffffc06c0445>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
  kernel: [<ffffffff8ce44286>] prune_super+0xf6/0x190
  kernel: [<ffffffff8cdc9de5>] shrink_slab+0x175/0x340
  kernel: [<ffffffff8cdbc8ef>] ? zone_watermark_ok+0x1f/0x30
  kernel: [<ffffffff8cdddf83>] ? compaction_suitable+0xa3/0xb0
  kernel: [<ffffffff8cdcdbe1>] balance_pgdat+0x4b1/0x5e0
  kernel: [<ffffffff8cdcde83>] kswapd+0x173/0x440
  kernel: [<ffffffff8ccc2d40>] ? wake_up_atomic_t+0x30/0x30
  kernel: [<ffffffff8cdcdd10>] ? balance_pgdat+0x5e0/0x5e0
  kernel: [<ffffffff8ccc1c71>] kthread+0xd1/0xe0
  kernel: [<ffffffff8ccc1ba0>] ? insert_kthread_work+0x40/0x40
  kernel: [<ffffffff8d374c37>] ret_from_fork_nospec_begin+0x21/0x21
  kernel: [<ffffffff8ccc1ba0>] ? insert_kthread_work+0x40/0x40
  kernel: INFO: task kswapd1:230 blocked for more than 120 seconds.
  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  kernel: kswapd1         D ffff8ebc1de4a080     0   230      2 0x00000080
  kernel: Call Trace:
  kernel: [<ffffffff8d368b69>] schedule_preempt_disabled+0x29/0x70
  kernel: [<ffffffff8d366ab7>] __mutex_lock_slowpath+0xc7/0x1d0
  kernel: [<ffffffff8d365e9f>] mutex_lock+0x1f/0x2f
  kernel: [<ffffffffc06aeccc>] xfs_reclaim_inodes_ag+0x2dc/0x390 [xfs]
  kernel: [<ffffffff8cce2e98>] ? enqueue_task_fair+0x208/0x6c0
  kernel: [<ffffffff8ccd9f15>] ? sched_clock_cpu+0x85/0xc0
  kernel: [<ffffffff8ccd2ee0>] ? check_preempt_curr+0x80/0xa0
  kernel: [<ffffffff8ccd2f19>] ? ttwu_do_wakeup+0x19/0xe0
  kernel: [<ffffffff8ccd304f>] ? ttwu_do_activate+0x6f/0x80
  kernel: [<ffffffff8ccd6530>] ? try_to_wake_up+0x190/0x390
  kernel: [<ffffffff8ccd6745>] ? wake_up_process+0x15/0x20
  kernel: [<ffffffffc06afe33>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
  kernel: [<ffffffffc06c0445>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
  kernel: [<ffffffff8ce44286>] prune_super+0xf6/0x190
  kernel: [<ffffffff8cdc9de5>] shrink_slab+0x175/0x340
  kernel: [<ffffffff8ce376c1>] ? vmpressure+0x61/0x90
  kernel: [<ffffffff8cdcdbe1>] balance_pgdat+0x4b1/0x5e0
  kernel: [<ffffffff8cdcde83>] kswapd+0x173/0x440
  kernel: [<ffffffff8ccc2d40>] ? wake_up_atomic_t+0x30/0x30
  kernel: [<ffffffff8cdcdd10>] ? balance_pgdat+0x5e0/0x5e0
  kernel: [<ffffffff8ccc1c71>] kthread+0xd1/0xe0
  kernel: [<ffffffff8ccc1ba0>] ? insert_kthread_work+0x40/0x40
  kernel: [<ffffffff8d374c37>] ret_from_fork_nospec_begin+0x21/0x21
  kernel: [<ffffffff8ccc1ba0>] ? insert_kthread_work+0x40/0x40
  

• Why many processes getting blocked for more than 120 seconds ?

• Why kswapd fail to release cached pages to free up the memory for new processes ?
• Why new processes are not being created on node ?