System stall with timed out NVMe over TCP requests

Issue

System stall with timed out NVMe over TCP requests:

kernel: nvme nvme2: queue 0: timeout request 0x0 type 4
kernel: nvme nvme2: starting error recovery
kernel: nvme2: Keep Alive(0x18), Host Aborted Command (sct 0x3 / sc 0x71) 
kernel: nvme nvme2: failed nvme_keep_alive_end_io error=10
kernel: nvme nvme2: Reconnecting in 10 seconds...
kernel: nvme nvme12: queue 0: timeout request 0x0 type 4
kernel: nvme nvme12: starting error recovery
kernel: nvme12: Keep Alive(0x18), Host Aborted Command (sct 0x3 / sc 0x71) 
kernel: nvme nvme12: failed nvme_keep_alive_end_io error=10
....
kernel: block nvme1n1: no usable path - requeuing I/O
kernel: block nvme1n1: no usable path - requeuing I/O
kernel: block nvme1n1: no usable path - requeuing I/O
kernel: block nvme1n1: no usable path - requeuing I/O
....
CPU: 33 PID: 1766 Comm: kworker/33:1H Kdump: loaded Tainted: GF         IOE    --------- ---  5.14.0-162.6.1.el9_1.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940xa/08XR9M, BIOS 2.16.1 08/19/2022
Workqueue: kblockd blk_mq_timeout_work
Call Trace:
 dump_stack_lvl+0x34/0x48
 panic+0x102/0x2d4
 probe_21069.cold+0x27/0x27 [stap_8895f793998afc5350ac68a146588f12_7643]
 ? nvme_tcp_timeout+0x5c/0xca [nvme_tcp]
 ? nvme_tcp_timeout+0x5b/0xca [nvme_tcp]
 enter_kprobe_probe+0x13a/0x2e0 [stap_8895f793998afc5350ac68a146588f12_7643]
 aggr_pre_handler+0x3d/0x80
 ? nvme_tcp_timeout+0x5c/0xca [nvme_tcp]
 kprobe_int3_handler+0x95/0x180
 do_int3+0x3b/0x80
 exc_int3+0x81/0xc0
 asm_exc_int3+0x35/0x40
RIP: 0010:nvme_tcp_timeout+0x5b/0xca [nvme_tcp]
....
 ? nvme_tcp_timeout+0x5c/0xca [nvme_tcp]
 ? nvme_tcp_timeout+0x5b/0xca [nvme_tcp]
 blk_mq_check_expired+0x71/0xb0
 bt_iter+0x97/0xb0
 blk_mq_queue_tag_busy_iter+0x2b8/0x5b0
 ? __blk_mq_complete_request_remote+0x10/0x10
 ? __blk_mq_complete_request_remote+0x10/0x10
 ? init_cfs_bandwidth+0x40/0x90
 blk_mq_timeout_work+0x5c/0x140