dm-multipath devices become corrupted and point to the same paths on Red Hat Enterprise Linux 6
Issue
-
When nodes get rebooted, the following errors are observed:
- When running lvm commands, "duplicate PV's" are found and both are multipath devices:
Found duplicate PV BP75I3rHEkyLNynrFwSgS95fGNdTvCCo: using `/dev/mapper/mpathdp1` not `/dev/mapper/mpathgp1`- The output of
multipath -llshows that there is more than 1 multipath device with the same paths (in this case,mpathdandmpathgboth reportsdkandsdeas their paths):
# multipath -ll <... some data omitted ...> mpathd (360060e801013b1d0058b3aed00000097) dm-5 HITACHI,DF600F size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active | `- 3:0:0:5 sdk 8:160 active ready running `-+- policy='round-robin 0' prio=0 status=enabled `- 4:0:0:5 sde 8:64 active ready running ... mpathg (360060e801013b1d0058b3aed00000069) dm-0 HITACHI,DF600F size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=1 status=active | `- 3:0:0:5 sdk 8:160 active ready running `-+- policy='round-robin 0' prio=0 status=enabled `- 4:0:0:5 sde 8:64 active ready running- Some other LVM commands report that they cannot find certain LVM Physical Volumes (PV's):
# pvscan Couldn't find device with uuid WZufCX-LdeX-jEQO-heUb-yAY0-y5IK-pmYlFX. Couldn't find device with uuid BP75I3-rHEk-yLNy-nrFw-SgS9-5fGN-dTvCCo. Couldn't find device with uuid WZufCX-LdeX-jEQO-heUb-yAY0-y5IK-pmYlFX. Couldn't find device with uuid BP75I3-rHEk-yLNy-nrFw-SgS9-5fGN-dTvCCo. There are 2 physical volumes missing. PV unknown device VG dbvg00 lvm2 [100.00 GiB / 1.99 GiB free] PV /dev/mapper/mpathhp1 VG dbvg00 lvm2 [100.00 GiB / 0 free] PV unknown device VG dbvg00 lvm2 [100.00 GiB / 0 free] PV /dev/mapper/mpathip1 VG dbvg00 lvm2 [100.00 GiB / 85.00 GiB free]
Environment
- Red Hat Enterprise Linux 6 (RHEL6)
- Observed on
kernel-2.6.32-279.el6, but it is currently unknown if other versions are also affected.
- Observed on
device-mapper-multipathmanaging SAN storage- Observed on
device-mapper-multipath-0.4.9-56.el6, but currently unknown if other versions are also affected. - The multipath bindings file is located at
/etc/multipath/bindings, and is located on the root filesystem (this is opposite to this issue). - Must be using
user_friendly_names yesoption in/etc/multipath.conffor this issue to occur.
- Observed on
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
