System hang due to stalled iSERT completions

Issue

• Observing hung tasks in the messages log:

kernel: INFO: task iscsi_np:3168 blocked for more than 120 seconds.
kernel:      Tainted: G           OE    --------- -  - 4.18.0-305.el8.x86_64 #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: iscsi_np        D    0  3168      2 0x80004084
kernel: Call Trace:
kernel: __schedule+0x2c4/0x700
kernel: schedule+0x38/0xa0
kernel: schedule_timeout+0x246/0x2f0
kernel: ? kfree+0xc3/0x450
kernel: ? iscsi_decode_text_input+0x314/0x840 [iscsi_target_mod]
kernel: wait_for_completion+0x97/0x100
kernel: iscsi_check_for_session_reinstatement+0x1e4/0x280 [iscsi_target_mod]
kernel: iscsi_target_do_login+0x21d/0x570 [iscsi_target_mod]
kernel: iscsi_target_start_negotiation+0x51/0xc0 [iscsi_target_mod]
kernel: iscsi_target_login_thread+0x820/0xe10 [iscsi_target_mod]
kernel: ? iscsi_target_login_sess_out+0x150/0x150 [iscsi_target_mod]
kernel: kthread+0x116/0x130
kernel: ? kthread_flush_work_fn+0x10/0x10
kernel: ret_from_fork+0x1f/0x40
kernel: INFO: task iscsi_ttx:4070 blocked for more than 120 seconds.
kernel:      Tainted: G           OE    --------- -  - 4.18.0-305.el8.x86_64 #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: iscsi_ttx       D    0  4070      2 0x80004084
kernel: Call Trace:
kernel: __schedule+0x2c4/0x700
kernel: schedule+0x38/0xa0
kernel: schedule_timeout+0x246/0x2f0
kernel: ? mlx5_ib_post_send+0x513/0x14f0 [mlx5_ib]
kernel: ? mlx5_ib_post_send+0x411/0x14f0 [mlx5_ib]
kernel: wait_for_completion+0x97/0x100
kernel: mlx5_ib_drain_sq+0x13b/0x160 [mlx5_ib]
kernel: ? mlx5_ib_destroy_dct+0x90/0x90 [mlx5_ib]
kernel: ib_drain_sq+0x1b/0x70 [ib_core]
kernel: ib_drain_qp+0xe/0x20 [ib_core]
kernel: isert_wait_conn+0x65/0x2b0 [ib_isert]
kernel: ? iscsit_thread_get_cpumask+0x70/0x70 [iscsi_target_mod]
kernel: iscsit_close_connection+0x15e/0x910 [iscsi_target_mod]
kernel: ? __schedule+0x2cc/0x700
kernel: ? iscsit_thread_get_cpumask+0x70/0x70 [iscsi_target_mod]
kernel: iscsit_take_action_for_connection_exit+0x7e/0x100 [iscsi_target_mod]
kernel: iscsi_target_tx_thread+0x174/0x1f0 [iscsi_target_mod]

• And the workqueue has busy task hung for a long time due to unavailable iscsit tag:

[153718.660251] kworker/119:2H  P    0 24013      2 0x80004080
[153718.660259] Workqueue: ib-comp-wq ib_cq_poll_work [ib_core]
[153718.660262] Call Trace:
[153718.660263]  __schedule+0x2c4/0x700
[153718.660265]  ? sbitmap_get+0xef/0x140
[153718.660266]  schedule+0x38/0xa0
[153718.660272]  iscsit_allocate_cmd+0x227/0x250 [iscsi_target_mod]
[153718.660274]  ? finish_wait+0x80/0x80
[153718.660277]  isert_allocate_cmd+0x1d/0x50 [ib_isert]
[153718.660279]  isert_recv_done+0x2a4/0x7a0 [ib_isert]
[153718.660286]  __ib_process_cq+0x91/0x1c0 [ib_core]
[153718.660292]  ib_cq_poll_work+0x26/0x80 [ib_core]
[153718.660294]  process_one_work+0x1a7/0x360
[153718.660296]  ? create_worker+0x1a0/0x1a0
[153718.660296]  worker_thread+0x30/0x390
[153718.660297]  ? create_worker+0x1a0/0x1a0
[153718.660298]  kthread+0x116/0x130
[153718.660299]  ? kthread_flush_work_fn+0x10/0x10
[153718.660301]  ret_from_fork+0x1f/0x40