Virtual Machine loses access to its disks after hot-plugging some extra disks.
Issue
- OpenShift worker nodes running on OpenStack are hanging and getting corrupted.
- Hosted Cluster (Hosted Control Planes) nodes regularly hang when running on OpenShift Virtualization
- The OS Disk of the Virtual Machine stops working after hotplug of some other extra disk:
[ 272.879151] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[ 283.104067] sd 0:0:0:0: [sda] tag#112 abort
[ 283.109673] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[ 283.122063] sd 0:0:0:0: device reset
[ 293.344050] sd 0:0:0:0: [sda] tag#112 abort
[ 293.348559] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 293.354367] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 293.360096] scsi_io_completion_action: 2 callbacks suppressed
[ 293.360099] sd 0:0:0:0: [sda] tag#113 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=20s
[ 293.372435] sd 0:0:0:0: [sda] tag#113 CDB: Write(10) 2a 00 00 a3 04 58 00 00 38 00
[ 293.378486] print_req_error: 10 callbacks suppressed
[ 293.378487] blk_update_request: I/O error, dev sda, sector 10683480 op 0x1:(WRITE) flags 0x100000 phys_seg 7 prio class 0
[ 293.391906] sd 0:0:0:0: rejecting I/O to offline device
[ 293.402309] blk_update_request: I/O error, dev sda, sector 5850816 op 0x1:(WRITE) flags 0x100000 phys_seg 42 prio class 0
[ 293.420761] blk_update_request: I/O error, dev sda, sector 10683440 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[ 293.439903] sda3: writeback error on inode 16912535, offset 4096, sector 10683448
[ 293.439910] sda3: writeback error on inode 16912535, offset 8192, sector 10683536
[ 293.448109] sda3: writeback error on inode 8900865, offset 221184, sector 5850824
[ 293.455655] sda3: writeback error on inode 8900865, offset 225280, sector 5851168
[ 293.497802] blk_update_request: I/O error, dev sda, sector 10607666 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 0
[ 293.515957] XFS (sda3): log I/O error -5
[ 293.519904] XFS (sda3): Log I/O error (0x2) detected at xlog_ioend_work+0x74/0x80 [xfs] (fs/xfs/xfs_log.c:1284). Shutting down filesystem
[ 293.530178] XFS (sda3): Please unmount the filesystem and rectify the problem(s)
Environment
- Red Hat Enterprise Linux (RHEL) 9.2
- Red Hat OpenStack Platform (RHOSP) 17.1
- Red Hat OpenShift Container Platform (RHOCP) 4.13, 4.14
- OpenShift Virtualization
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.