System hangs during NVMe reset following I/O issues

Solution Verified - Updated -

Issue

  • System hangs during NVMe reset following I/O issues
[3503230.259838] nvme nvme1: I/O 23 QID 2 timeout, aborting
[3503230.265461] nvme nvme1: I/O 24 QID 2 timeout, aborting
[3503232.505179] nvme nvme1: Abort status: 0x2
[3503232.513360] nvme nvme1: Abort status: 0x2
[3503275.268119] nvme nvme1: I/O 29 QID 1 timeout, aborting
[3503275.273657] nvme nvme1: I/O 30 QID 1 timeout, aborting
[3503275.279126] nvme nvme1: I/O 31 QID 1 timeout, aborting
[3503276.256124] nvme nvme1: I/O 17 QID 2 timeout, aborting
[3503276.261700] nvme nvme1: I/O 18 QID 2 timeout, aborting
[3503302.233470] nvme nvme1: Abort status: 0x2
[3503302.238701] nvme nvme1: Abort status: 0x2
[3503302.244170] nvme nvme1: Abort status: 0x2
[3503302.249889] nvme nvme1: Abort status: 0x2
[3503302.255043] nvme nvme1: Abort status: 0x2
[3503341.316553] nvme nvme1: I/O 9 QID 2 timeout, aborting
[3503341.322102] nvme nvme1: I/O 10 QID 2 timeout, aborting
[3503341.327633] nvme nvme1: I/O 11 QID 2 timeout, aborting
[3503342.310557] nvme nvme1: I/O 6 QID 1 timeout, aborting
[3503342.316094] nvme nvme1: I/O 7 QID 1 timeout, aborting
[3503372.324752] nvme nvme1: I/O 6 QID 1 timeout, reset controller
....
[3503372.518439] blk_update_request: I/O error, dev nvme1n1, sector 18454357024
[3503372.524821] blk_update_request: I/O error, dev nvme1n1, sector 18454357536
[3503372.531301] blk_update_request: I/O error, dev nvme1n1, sector 7749786480
[3503372.543035] nvme 0000:00:09.0: irq 91 for MSI/MSI-X
[3503372.645400] nvme 0000:00:09.0: irq 91 for MSI/MSI-X
[3503372.645421] nvme 0000:00:09.0: irq 104 for MSI/MSI-X
[3503372.645449] nvme 0000:00:09.0: irq 105 for MSI/MSI-X
[3503403.256952] nvme nvme1: I/O 0 QID 2 timeout, disable controller
[3503403.380497] blk_update_request: 70 callbacks suppressed
[3503403.386084] blk_update_request: I/O error, dev nvme1n1, sector 2005584448
[3503403.392365] blk_update_request: I/O error, dev nvme1n1, sector 19422768432
[3503403.398654] blk_update_request: I/O error, dev nvme1n1, sector 9554735408
....
[3503668.562836] Kernel panic - not syncing: cxiPanic: thread 65071 pid 63597: forced panic: Forcing kernel panic to clear hung I/O

Environment

  • Red Hat Enterprise Linux 7.9

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content