dm-multipath devices become corrupted and point to the same paths on Red Hat Enterprise Linux 6

Issue

• When nodes get rebooted, the following errors are observed:

  • When running lvm commands, "duplicate PV's" are found and both are multipath devices:

  Found duplicate PV BP75I3rHEkyLNynrFwSgS95fGNdTvCCo: using `/dev/mapper/mpathdp1` not `/dev/mapper/mpathgp1`
  

  • The output of multipath -ll shows that there is more than 1 multipath device with the same paths (in this case, mpathd and mpathg both report sdk and sde as their paths):

  # multipath -ll
  <... some data omitted ...>
  mpathd (360060e801013b1d0058b3aed00000097) dm-5 HITACHI,DF600F
  size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
  |-+- policy='round-robin 0' prio=1 status=active
  | `- 3:0:0:5   sdk 8:160 active ready running
  `-+- policy='round-robin 0' prio=0 status=enabled
    `- 4:0:0:5   sde 8:64  active ready running
  ...
  mpathg (360060e801013b1d0058b3aed00000069) dm-0 HITACHI,DF600F
  size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
  |-+- policy='round-robin 0' prio=1 status=active
  | `- 3:0:0:5   sdk 8:160 active ready running
  `-+- policy='round-robin 0' prio=0 status=enabled
    `- 4:0:0:5   sde 8:64  active ready running
  

  • Some other LVM commands report that they cannot find certain LVM Physical Volumes (PV's):

  # pvscan
    Couldn't find device with uuid WZufCX-LdeX-jEQO-heUb-yAY0-y5IK-pmYlFX.
    Couldn't find device with uuid BP75I3-rHEk-yLNy-nrFw-SgS9-5fGN-dTvCCo.
      Couldn't find device with uuid WZufCX-LdeX-jEQO-heUb-yAY0-y5IK-pmYlFX.
      Couldn't find device with uuid BP75I3-rHEk-yLNy-nrFw-SgS9-5fGN-dTvCCo.
      There are 2 physical volumes missing.
    PV unknown device          VG dbvg00     lvm2 [100.00 GiB / 1.99 GiB free]
    PV /dev/mapper/mpathhp1    VG dbvg00     lvm2 [100.00 GiB / 0    free]
    PV unknown device          VG dbvg00     lvm2 [100.00 GiB / 0    free]
    PV /dev/mapper/mpathip1    VG dbvg00     lvm2 [100.00 GiB / 85.00 GiB free]