Seeing kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH and path failures

Issue

• Cisco UCS blades and Nexus switches with mismatching QoS settings experiencing FC underruns under heavy load.
• SCSI timeouts and multipath path failures observed in /var/log/messages.

• The UCS switch logs show:

   %IPQOSMGR-2-QOSMGR_DCBXP_PFC_CMP_FAIL_MSG:  Ethernet1/31 - qos config for PFC not compatible with the peer
   %IPQOSMGR-2-QOSMGR_DCBXP_PFC_CMP_FAIL_MSG:  Ethernet1/30 - qos config for PFC not compatible with the peer
  

• The system will log

  kernel: scsi host1: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host1: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host1: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host1: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host2: hdr status = FCPIO_DATA_CNT_MISMATCH
  kernel: scsi host1: hdr status = FCPIO_DATA_CNT_MISMATCH
  

• And:

  kernel: sd 2:0:0:16: [sdcy] Medium access timeout failure. Offlining disk!
  kernel: device-mapper: multipath: Failing path 70:96.
  multipathd: checker failed path 70:96 in map mpathq
  multipathd: mpathq: remaining active paths: 3
  kernel: host2: Assigned Port ID 7c0008
  kernel: sd 2:0:0:16: [sdcy]  Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
  kernel: sd 2:0:0:16: [sdcy] CDB: Read(10): 28 00 04 96 2c 00 00 04 00 00
  kernel: end_request: I/O error, dev sdcy, sector 76950528
  kernel: sd 2:0:0:16: rejecting I/O to offline device
  kernel: sd 2:0:0:16: [sdcy] killing request
  kernel: sd 2:0:0:16: rejecting I/O to offline device
  kernel: sd 2:0:0:16: rejecting I/O to offline device
  kernel: sd 2:0:0:16: rejecting I/O to offline device
  kernel: sd 2:0:0:16: rejecting I/O to offline device
  kernel: sd 2:0:0:16: [sdcy]  Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
  kernel: sd 2:0:0:16: [sdcy] CDB: Read(10): 28 00 03 2f cc 00 00 04 00 00
  kernel: end_request: I/O error, dev sdcy, sector 53464064
  kernel: sd 2:0:0:16: [sdcy]  Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
  kernel: sd 2:0:0:16: [sdcy] CDB: Read(10): 28 00 00 09 fc 00 00 04 00 00
  kernel: end_request: I/O error, dev sdcy, sector 654336