'nvme format' command hits timeout and hangs.

Issue

• nvme format command hits timeout and hangs:

[  842.635742] block nvme1n1: No UUID available providing old NGUID
[  842.662610] block nvme1n1: No UUID available providing old NGUID
[  946.950734] nvme nvme2: I/O 18 QID 0 timeout, reset controller
[  947.021006] nvme2: Format NVM(0x80), Host Aborted Command (sct 0x3 / sc 0x71) 
[ 1110.783233] INFO: task nvme:90341 blocked for more than 122 seconds.
[ 1110.790331]       Tainted: G           OE    --------  ---  5.14.0-284.25.1.el9_2.x86_64 #1
[ 1110.799654] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1110.808392] task:nvme            state:D stack:    0 pid:90341 ppid: 30957 flags:0x00004002
[ 1110.817720] Call Trace:
[ 1110.820448]  <TASK>
[ 1110.822790]  __schedule+0x248/0x620
[ 1110.826686]  schedule+0x5a/0xc0
[ 1110.830189]  schedule_timeout+0x11d/0x160
[ 1110.834663]  ? select_idle_sibling+0x28/0x520
[ 1110.839526]  ? schedule_timeout+0x90/0x160
[ 1110.844101]  ? wake_affine+0x62/0x1f0
[ 1110.848195]  __wait_for_common+0x93/0x1d0
[ 1110.852672]  ? usleep_range_state+0x90/0x90
[ 1110.857340]  __flush_work.isra.0+0x160/0x230
[ 1110.862104]  ? flush_workqueue_prep_pwqs+0x110/0x110
[ 1110.867648]  nvme_execute_passthru_rq+0x1a6/0x270 [nvme_core]
[ 1110.874069]  nvme_submit_user_cmd+0x1d8/0x250 [nvme_core]
[ 1110.880097]  nvme_user_cmd+0x138/0x1b0 [nvme_core]
[ 1110.885448]  blkdev_ioctl+0x151/0x290
[ 1110.889537]  ? security_file_ioctl+0x32/0x50
[ 1110.894303]  __x64_sys_ioctl+0x8a/0xc0
[ 1110.898488]  do_syscall_64+0x5c/0x90
[ 1110.902478]  ? syscall_exit_work+0x11a/0x150
[ 1110.907242]  ? syscall_exit_to_user_mode+0x12/0x30
[ 1110.912589]  ? do_syscall_64+0x69/0x90
[ 1110.916772]  ? syscall_exit_to_user_mode+0x12/0x30
[ 1110.922117]  ? do_syscall_64+0x69/0x90
[ 1110.926305]  ? syscall_exit_to_user_mode+0x12/0x30
[ 1110.931654]  ? do_syscall_64+0x69/0x90
[ 1110.935840]  ? exc_page_fault+0x62/0x150
[ 1110.940217]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 1110.945858] RIP: 0033:0x7f7919c3ec6b
[ 1110.949848] RSP: 002b:00007ffdd8ab6648 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[ 1110.958290] RAX: ffffffffffffffda RBX: 00007ffdd8ab6670 RCX: 00007f7919c3ec6b
[ 1110.966253] RDX: 00007ffdd8ab6670 RSI: 00000000c0484e41 RDI: 0000000000000003
[ 1110.974217] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f7919db14e0
[ 1110.982187] R10: 00007ffdd8ab6690 R11: 0000000000000206 R12: 0000000000000000
[ 1110.990152] R13: 00007ffdd8ab66e0 R14: 00007ffdd8ab6740 R15: 0000000000000003
[ 1110.998119]  </TASK>

• Command similar to the following was run:

# date; nvme --format /dev/nvme2n1 --lbaf=0

• List of NVMe devices from the system. Issue reproduced on TBu NVMe drive.

Node             SN                   Model                                    Namespace Capacity(gendisk)  Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- ------------------ ---------------- --------
/dev/nvme0n1     22343AF87DD2         Micron_7450_MTFDKBZ960TFR                1         894 GiB            512   B + 0 B    E2MU110 
/dev/nvme1n1     22343AF87CD8         Micron_7450_MTFDKBZ960TFR                1         894 GiB            512   B + 0 B    E2MU110 
/dev/nvme2n1     03JN351YS32BG26R00C  38.4 TBu FCM4 NVMe G4                    1         838 TiB            4   KiB + 0 B    4_0_2