RHEL8.7 fnic driver returning SCSI_MLQUEUE_HOST_BUSY and stalling any commands for long periods of time

Issue

• kworker process is seen in blocked state trying to remove a device
• System load rises after path is lost on multiple multipath devices and the system gets very slow and unresponsive

• From the logs we see multipath fail paths through the stalled ports due to TUR path_checker timeouts.

  multipathd[1687]: xxx_1: remaining active paths: 3
  multipathd[1687]: xxx_1: remaining active paths: 2     
  

• And after 2 minutes we see the targets removed

  kernel: rport-8:0-1: blocked FC remote port time out: removing target and saving binding
  kernel: sd 8:0:1:0: [sde] Synchronizing SCSI cache
  kernel: sd 8:0:1:0: Device offlined - not ready after error recovery
  kernel: sd 8:0:1:0: [sde] tag#213 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT cmd_age=761s
  kernel: sd 8:0:1:0: [sde] tag#213 CDB: Test Unit Ready 00 00 00 00 00 00