RHEL 8.8/8.6(EUS): hung_task_timeout_secs at migration_entry_wait_on_locked

Issue

• After upgrading to RHEL 8.8, few commands are going to hung state and system load average showing very high.
• Logs shows several hung_task_timeout_secs with migration_entry_wait_on_locked() in backtrace.

INFO: task task1:1618 blocked for more than 120 seconds.
      Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:task1     state:D stack:    0 pid: 1618 ppid:     1 flags:0x00000080
Call Trace:
 __schedule+0x2d1/0x870
 schedule+0x55/0xf0
 io_schedule+0x12/0x40
 migration_entry_wait_on_locked+0x1ea/0x290
 do_swap_page+0x5b0/0x710
 __handle_mm_fault+0x453/0x6c0
 handle_mm_fault+0xca/0x2a0
 __do_page_fault+0x1f0/0x450
 do_page_fault+0x37/0x130
 page_fault+0x1e/0x30

• After upgrading to RHEL 8.6 EUS kernel-4.18.0-372.91.1.el8_6, few commands are going to hung state and system load average showing very high.
• Logs shows several hung_task_timeout_secs with migration_entry_wait_on_locked() in backtrace.

INFO: task task2:2159306 blocked for more than 120 seconds.
      Tainted: GOE  ----------- 4.18.0-372.91.1.el8_6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:task2  state:D stack:    0 pid:2159306 ppid:     1 flags:0x00000084
Call Trace:
 __schedule+0x2d1/0x860
 schedule+0x55/0xf0
 io_schedule+0x12/0x40
 migration_entry_wait_on_locked+0x1e0/0x280
 do_swap_page+0x5b0/0x710
 __handle_mm_fault+0x453/0x6d0
 handle_mm_fault+0xca/0x2a0
 __do_page_fault+0x1d0/0x420
 do_page_fault+0x37/0x12d
 page_fault+0x1e/0x30

• A hard reset might be required as the issue can prevent the system from rebooting gracefully.