System unresponsive even though the each running task shows running in application context

Issue

• The system had high load average and not responding.
• Call trace shows the below and each runqueue was running application in process context.

[62685.428148] INFO: task kworker/0:3:234676 blocked for more than 120 seconds.
[62685.428156]       Not tainted 4.18.0-348.2.1.el8_5.x86_64 #1
[62685.428157] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[62685.428158] task:kworker/0:3     state:D stack:    0 pid:234676 ppid:     2 flags:0x80004080
[62685.428182] Workqueue: events key_garbage_collector
[62685.428183] Call Trace:
[62685.428199]  __schedule+0x2c4/0x700
[62685.428202]  schedule+0x37/0xa0
[62685.428204]  schedule_timeout+0x274/0x300
[62685.428209]  ? __blk_mq_free_request+0x6b/0x90
[62685.428210]  ? __switch_to_asm+0x41/0x70
[62685.428211]  ? __switch_to_asm+0x35/0x70
[62685.428212]  ? __switch_to_asm+0x41/0x70
[62685.428212]  ? __switch_to_asm+0x35/0x70
[62685.428225]  ? insert_work+0x65/0xb0
[62685.428234]  ? next_online_pgdat+0x22/0x50
[62685.428235]  wait_for_completion+0x97/0x100
[62685.428244]  __wait_rcu_gp+0x10a/0x130
[62685.428248]  synchronize_rcu+0x76/0x80
[62685.428249]  ? rcu_accelerate_cbs_unlocked+0x70/0x70
[62685.428250]  ? __bpf_trace_rcu_utilization+0x10/0x10
[62685.428252]  key_garbage_collector+0x1d7/0x3e0
[62685.428254]  process_one_work+0x1a7/0x360
[62685.428256]  worker_thread+0x30/0x390
[62685.428258]  ? create_worker+0x1a0/0x1a0
[62685.428263]  kthread+0x116/0x130
[62685.428264]  ? kthread_flush_work_fn+0x10/0x10
[62685.428265]  ret_from_fork+0x1f/0x40
[64775.855314] INFO: task kworker/0:1:1164971 blocked for more than 120 seconds.
[64775.855321]       Not tainted 4.18.0-348.2.1.el8_5.x86_64 #1
[64775.855322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[64775.855325] task:kworker/0:1     state:D stack:    0 pid:1164971 ppid:     2 flags:0x80004080
[64775.855333] Workqueue: events vmci_transport_cleanup [vmw_vsock_vmci_transport]
[64775.855335] Call Trace:
[64775.855345]  __schedule+0x2c4/0x700
[64775.855347]  schedule+0x37/0xa0
[64775.855350]  schedule_timeout+0x274/0x300
[64775.855352]  ? __switch_to_asm+0x41/0x70
[64775.855353]  ? __switch_to_asm+0x35/0x70
[64775.855354]  wait_for_completion+0x97/0x100
[64775.855360]  __wait_rcu_gp+0x10a/0x130
[64775.855363]  synchronize_rcu+0x76/0x80
[64775.855364]  ? rcu_accelerate_cbs_unlocked+0x70/0x70
[64775.855365]  ? __bpf_trace_rcu_utilization+0x10/0x10
[64775.855370]  vmci_event_unsubscribe+0x80/0xb0 [vmw_vmci]
[64775.855372]  vmci_transport_free_resources+0x66/0xc0 [vmw_vsock_vmci_transport]
[64775.855374]  vmci_transport_cleanup+0x75/0x90 [vmw_vsock_vmci_transport]
[64775.855378]  process_one_work+0x1a7/0x360
[64775.855380]  worker_thread+0x30/0x390
[64775.855382]  ? create_worker+0x1a0/0x1a0
[64775.855385]  kthread+0x116/0x130
[64775.855386]  ? kthread_flush_work_fn+0x10/0x10
[64775.855387]  ret_from_fork+0x1f/0x40