Virtual Machine loses access to its disks after hot-plugging some extra disks.

Solution Verified - Updated -

Issue

  • OpenShift worker nodes running on OpenStack are hanging and getting corrupted.
  • Hosted Cluster (Hosted Control Planes) nodes regularly hang when running on OpenShift Virtualization
  • The OS Disk of the Virtual Machine stops working after hotplug of some other extra disk:
[  272.879151] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  283.104067] sd 0:0:0:0: [sda] tag#112 abort
[  283.109673] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  283.122063] sd 0:0:0:0: device reset
[  293.344050] sd 0:0:0:0: [sda] tag#112 abort
[  293.348559] sd 0:0:0:0: Device offlined - not ready after error recovery
[  293.354367] sd 0:0:0:0: Device offlined - not ready after error recovery
[  293.360096] scsi_io_completion_action: 2 callbacks suppressed
[  293.360099] sd 0:0:0:0: [sda] tag#113 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=20s
[  293.372435] sd 0:0:0:0: [sda] tag#113 CDB: Write(10) 2a 00 00 a3 04 58 00 00 38 00
[  293.378486] print_req_error: 10 callbacks suppressed
[  293.378487] blk_update_request: I/O error, dev sda, sector 10683480 op 0x1:(WRITE) flags 0x100000 phys_seg 7 prio class 0
[  293.391906] sd 0:0:0:0: rejecting I/O to offline device
[  293.402309] blk_update_request: I/O error, dev sda, sector 5850816 op 0x1:(WRITE) flags 0x100000 phys_seg 42 prio class 0
[  293.420761] blk_update_request: I/O error, dev sda, sector 10683440 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[  293.439903] sda3: writeback error on inode 16912535, offset 4096, sector 10683448
[  293.439910] sda3: writeback error on inode 16912535, offset 8192, sector 10683536
[  293.448109] sda3: writeback error on inode 8900865, offset 221184, sector 5850824
[  293.455655] sda3: writeback error on inode 8900865, offset 225280, sector 5851168
[  293.497802] blk_update_request: I/O error, dev sda, sector 10607666 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 0
[  293.515957] XFS (sda3): log I/O error -5
[  293.519904] XFS (sda3): Log I/O error (0x2) detected at xlog_ioend_work+0x74/0x80 [xfs] (fs/xfs/xfs_log.c:1284). Shutting down filesystem
[  293.530178] XFS (sda3): Please unmount the filesystem and rectify the problem(s)

Environment

  • Red Hat Enterprise Linux (RHEL) 9.2
  • Red Hat OpenStack Platform (RHOSP) 17.1
  • Red Hat OpenShift Container Platform (RHOCP) 4.13, 4.14
    • OpenShift Virtualization

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content