Why are SCSI write errors showing up on gluster-block devices?

Issue

• In the messages file of an OCP node, there are continuous SCSI write errors pointing to a gluster-block disk device:

  Sep 27 03:27:10 host1 kernel: XFS (sdb): Failing async write on buffer block 0x1. Retrying async write.
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a1 d3 28 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 20 18 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 f0 c7 88 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a0 55 88 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a1 cf a8 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 1a 38 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a1 cf e0 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 f0 c5 c0 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a1 cf b0 00 00 08 00
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] Add. Sense: Write error
  Sep 27 03:27:13 host1 kernel: sd 109:0:0:0: [sda] CDB: Write(10) 2a 00 00 a0 26 b5 00 00 07 00
  Sep 27 03:27:22 host1 kernel: sd 163:0:0:0: [sdb] Add. Sense: Write error
  Sep 27 03:27:22 host1 kernel: sd 163:0:0:0: [sdb] CDB: Write(10) 2a 00 00 00 00 08 00 00 10 00
  Sep 27 03:27:22 host1 kernel: XFS (sdb): Failing async write on buffer block 0x8. Retrying async write.
  Sep 27 03:27:22 host1 kernel: XFS (sdb): Failing async write on buffer block 0x10. Retrying async write.
  

  In the example above, these messages are showing up for disks sdb and sda. These are two of the three available paths for the multipathed device mpatha:

  multipath -ll
  
  mpatha (36001405ef316567bd45440cae3aa58f9) dm-17 LIO-ORG ,TCMU device
  size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | `- 3:0:0:0 sdb 8:16 active ready running
  |-+- policy='round-robin 0' prio=-1 status=enabled
  | `- 2:0:0:0 sda 8:0  active ready running
  |`-+- policy='round-robin 0' prio=-1 status=enabled
  |  `- 4:0:0:0 sdc 8:32 active ready running
  

  This device is a persistent gluster-block volume:

  mount -l | grep -i mpath
  
   /dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.74.176.231:3260-iqn.2016-12.org.gluster-block:ef316567-bd45-440c-ae3a-a58f922b2229-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
  /dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/pods/cedd7de2-1a9d-11eb-8bed-fa163eea5134/volumes/kubernetes.io~iscsi/pvc-c74d8637-1a9d-11eb-8bed-fa163eea5134 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
  

  From the output above, this disk maps to persistent volume pvc-c74d8637-1a9d-11eb-8bed-fa163eea5134 in Openshift.

• Why are these errors observed? How to fix this issue?