System hangs during NVMe reset following I/O issues
Issue
- System hangs during NVMe reset following I/O issues
[3503230.259838] nvme nvme1: I/O 23 QID 2 timeout, aborting
[3503230.265461] nvme nvme1: I/O 24 QID 2 timeout, aborting
[3503232.505179] nvme nvme1: Abort status: 0x2
[3503232.513360] nvme nvme1: Abort status: 0x2
[3503275.268119] nvme nvme1: I/O 29 QID 1 timeout, aborting
[3503275.273657] nvme nvme1: I/O 30 QID 1 timeout, aborting
[3503275.279126] nvme nvme1: I/O 31 QID 1 timeout, aborting
[3503276.256124] nvme nvme1: I/O 17 QID 2 timeout, aborting
[3503276.261700] nvme nvme1: I/O 18 QID 2 timeout, aborting
[3503302.233470] nvme nvme1: Abort status: 0x2
[3503302.238701] nvme nvme1: Abort status: 0x2
[3503302.244170] nvme nvme1: Abort status: 0x2
[3503302.249889] nvme nvme1: Abort status: 0x2
[3503302.255043] nvme nvme1: Abort status: 0x2
[3503341.316553] nvme nvme1: I/O 9 QID 2 timeout, aborting
[3503341.322102] nvme nvme1: I/O 10 QID 2 timeout, aborting
[3503341.327633] nvme nvme1: I/O 11 QID 2 timeout, aborting
[3503342.310557] nvme nvme1: I/O 6 QID 1 timeout, aborting
[3503342.316094] nvme nvme1: I/O 7 QID 1 timeout, aborting
[3503372.324752] nvme nvme1: I/O 6 QID 1 timeout, reset controller
....
[3503372.518439] blk_update_request: I/O error, dev nvme1n1, sector 18454357024
[3503372.524821] blk_update_request: I/O error, dev nvme1n1, sector 18454357536
[3503372.531301] blk_update_request: I/O error, dev nvme1n1, sector 7749786480
[3503372.543035] nvme 0000:00:09.0: irq 91 for MSI/MSI-X
[3503372.645400] nvme 0000:00:09.0: irq 91 for MSI/MSI-X
[3503372.645421] nvme 0000:00:09.0: irq 104 for MSI/MSI-X
[3503372.645449] nvme 0000:00:09.0: irq 105 for MSI/MSI-X
[3503403.256952] nvme nvme1: I/O 0 QID 2 timeout, disable controller
[3503403.380497] blk_update_request: 70 callbacks suppressed
[3503403.386084] blk_update_request: I/O error, dev nvme1n1, sector 2005584448
[3503403.392365] blk_update_request: I/O error, dev nvme1n1, sector 19422768432
[3503403.398654] blk_update_request: I/O error, dev nvme1n1, sector 9554735408
....
[3503668.562836] Kernel panic - not syncing: cxiPanic: thread 65071 pid 63597: forced panic: Forcing kernel panic to clear hung I/O
Environment
- Red Hat Enterprise Linux 7.9
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.