RHEL9: The rcu-tasks, "rcu_tasks_kthre", blocked in UN stat for more than 120 seconds.

Issue

• Kernel panic when any process gets blocked for more than 120 seconds:

PANIC: "Kernel panic - not syncing: hung_task: blocked tasks"

crash> pd sysctl_hung_task_panic
sysctl_hung_task_panic = $1 = 1

• The rcu_tasks, "rcu_tasks_kthre", stuck in UN stat for more than 226 seconds:

crash> ps -m 12
[0 00:03:46.795] [UN]  PID: 12       TASK: ffff95ba403dba00  CPU: 1    COMMAND: "rcu_tasks_kthre"

PID: 12       TASK: ffff95ba403dba00  CPU: 1    COMMAND: "rcu_tasks_kthre"
 #0 [ffffbf0cc00e7cb0] __schedule at ffffffffb988fb7b
 #1 [ffffbf0cc00e7d18] schedule at ffffffffb989000d
 #2 [ffffbf0cc00e7d28] schedule_timeout at ffffffffb9895a7f
 #3 [ffffbf0cc00e7d90] __wait_for_common at ffffffffb9890b00
 #4 [ffffbf0cc00e7e08] __synchronize_srcu at ffffffffb8daa0e7
 #5 [ffffbf0cc00e7e68] rcu_tasks_postscan at ffffffffb8da60f5
 #6 [ffffbf0cc00e7e70] rcu_tasks_wait_gp at ffffffffb8da6440
 #7 [ffffbf0cc00e7ee0] rcu_tasks_one_gp at ffffffffb8da5f96
 #8 [ffffbf0cc00e7f00] rcu_tasks_kthread at ffffffffb8da614e
 #9 [ffffbf0cc00e7f18] kthread at ffffffffb8d358dd
#10 [ffffbf0cc00e7f50] ret_from_fork at ffffffffb8c02c69

• Kernel log has the following messages:

tasks_rcu_exit_srcu_stall: rcu_tasks grace period number 5 (since boot) gp_state: RTGS_POST_SCAN_TASKLIST is 225205 jiffies old.

Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish()