Why do I see 'Found duplicate PV' warnings when using LVM with multipath storage in RHEL?

Resolution

The most common cause of this issue is lvm claims the device before a different device mapper layer, such as multipath, can claim the device. Once lvm has the device open for exclusive access, the device is no longer available for use by multipath. There can be a race condition between lvm and multipath in which one gets to claim a device first as these can be proceeding in parallel during boot. Setting up a filter can prevent this by forcing lvm to only scan the devices the administrator desires.

• The two most common resolutions for this issue are:
  "Set up LVM filters" within /etc/lvm/lvm.conf to avoid scanning unnecessary devices.

  • In many cases setting up a filter in lvm.conf may not be required, but doing so ensures that only appropriate devices are scanned by lvm for use -- this includes rejecting all but local disks and accepting multipath devices instead of the default of all plain sd* and other similar disk devices.

  • There can be a race condition between lvm and multipath on which gets to claim exclusive access to a device first as these can be proceeding in parallel during boot. Setting up a filter adds guarantees that lvm will only scan and claim appropriate devices.

  • There are instances where multiple reboots without an explicit lvm filter works correctly with multipath until it suddenly does not on a following reboot. This is a reflection of the parallel discovery paths in lvm and multipath and the nature of timing that can affect which gets to claim exclusive access first. Again, setting of an explicit filter ensures lvm can only claim exclusive access to desired devices -- be it sd* or mpath.

• Rebuild the initramfs boot image file ↴ to ensure it has the most current copies of lvm and multipath configuration files along with making sure the boot image has the multipath modules present.

  • If not present, the multipath will be loaded after the boot pivot point which is after lvm has had a chance to view and claim all devices ahead of multipath. This is especially true if a filter is not in place.

Case 1: The two devices displayed in the output are both single paths to the same device:
Example:

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/sdd not /dev/sdf

Here, sdd and sdf can both be found under the same multipath map, in the multipath -ll output.

mpatha 360000000000000000000000000000000 dm-2 Vendor,Model
size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|- policy='service-time 0' prio=50 status=active
|- 2:0:0:0  sdd  8:48     active ready running
`- 3:0:0:0  sdf  65:240   active ready running

In this case, a filter can be configured in /etc/lvm/lvm.conf to restrict the devices that LVM will search for metadata. The filter is a list of patterns that will be applied to each device found by a scan of /dev (or the directory specified by the dir keyword in lvm.conf). Patterns are regular expressions delimited by any character and preceded by a (for accept) or r (for reject). The list is traversed in order, and the first regex that matches a device determines if the device will be accepted or rejected (ignored). Devices that don’t match any patterns are accepted.

This filter should include all devices that need to be checked for LVM metadata, such as the local hard drive with the root volume group on it and any multipath devices. By rejecting the underlying paths to a multipath device (such as /dev/sdb, /dev/sdd, etc) you can avoid these duplicate PV warnings since each unique metadata area will only be found once on the multipath device itself.

Example filters that will avoid duplicate PV warnings due to multiple storage paths being available:

• To accept the second partition on the first hard drive (/dev/sda) and any device-mapper-multipath devices, while rejecting everything else:

filter = [ "a|/dev/sda2$|", "a|/dev/mapper/mpath.*|", "r|.*|" ]

• To accept all HP SmartArray controllers and any EMC PowerPath devices:

filter = [ "a|/dev/cciss/.*|", "a|/dev/emcpower.*|", "r|.*|" ]

• To accept any partitions on the first IDE drive and any multipath devices:

filter = [ "a|/dev/hda.*|", "a|/dev/mapper/mpath.*|", "r|.*|" ]

NOTE: When adding a new filter to /etc/lvm/lvm.conf, ensure that the original filter is either commented out with a # or is removed.

Once a filter has been configured and lvm.conf saved, check the output of these commands to ensure that no physical volumes or volume groups are missing:

# pvscan
# vgscan

The filter can also be tested on the fly, without having to modify /etc/lvm/lvm.conf adding the --config argument to the LVM command. For example:

# lvs --config 'devices{ filter = [ "a|/dev/emcpower.*|", "r|.*|" ] }'

NOTE: Testing filters using the --config parameter will not make permanent changes to the server's configuration. Make sure to include the working filter in /etc/lvm/lvm.conf after testing.

Once the desired filter is configured, it is recommended to rebuild the initrd with mkinitrd (RHEL4 and 5) or dracut (RHEL 6 and 7 ) so that only the necessary devices are scanned upon reboot.

Case 2: The two devices are an multipath device and a path under the multipath device.
Example:

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjzzzz: using /dev/mapper/mpatha not /dev/sdd
Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjzzzz: using /dev/sdd not /dev/mapper/mpatha

Where mpatha in multipath -ll shows sdd as a path:

mpatha 360000000000000000000000000000000 dm-2 Vendor,Model
size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|- policy='service-time 0' prio=50 status=active
|- 2:0:0:0  sdd  8:48     active ready running
`- 3:0:0:0  sdf  65:240   active ready running

See Case 1 ⤵ as this is a variant of that issue with the same resolution of setting an lvm.conf filter to prevent scanning of PVs on both the multipath device and its underlying sdx paths.

Case 3: The two devices displayed in the output are both multipath maps.
Example:

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/mapper/mpatha not /dev/mapper/mpathc

Or

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/emcpowera not /dev/emcpowerh

Here, we are not looking at two different paths but two different devices. This is a lot more serious than the case above as it often means that the machine has been presented devices which it shouldn't be seeing (for example, LUN clones or mirrors). In this case, unless you have a clear idea of what devices should be removed from the machine, the situation could be unrecoverable. You might want to try contacting the Red Hat Technical Support anyway.

Case 4: The multipath modules and/or lvm, multipath configuration files are missing or out of date within the boot image file.

Found duplicate PV  ZC3kXXXXXXXXXXXXjd7ardDDMSMP: using /dev/sdx not /dev/sdy
Using duplicate PV  /dev/sdx which is last seen, replacing /dev/sdy
Found duplicate PV  ZC3kXXXXXXXXXXXXjd7ardDDMSMP: using /dev/sdx not /dev/sdy
  Using duplicate PV /dev/sdbq5 without holders, ignoring /dev/sdcx5
  WARNING: Device mismatch detected for xyzvg/tmpdir which is accessing /dev/sdx instead of /dev/sdy.
  WARNING: Device mismatch detected for xyzvg/var which is accessing /dev/sdx instead of /dev/sdy.

Issue: The difference between this case and case #1 is that while lvm.conf has appropriate filters set and/or multipath is configured -- the configuration files within the initial boot image file are older out-of-date versions of the configuration files and/or boot image file is missing needed multipath modules.

• Diagnostics:

  • Use lsinitrd command to verify the size and date of the lvm and multipath configuration files within the boot image.

    • Extract the initial boot image file to verify multipath modules are not missing. Refer to the article for the detailed steps on "How to extract the contents of initramfs image?"

• Rebuild initial boot image file:

  • Take a backup of the current boot image file.

  • Rebuild boot image file for the current kernel version to pick up latest lvm/multipath configuration files and ensure multipath modules present at boot time.

    • During instances like installing new kernel, multipath modules and configuration files might be missing in the initial boot image file due to recent, but post-install, configuration changes.

    • After any changes to configuration files that may be used at the early stage of the boot process, it's necessary to rebuild the initial boot image file to include the proper kernel modules, files, and configuration directives. Especially for proper lvm and multipath use.

• Refer to this article for detailed steps on "How to rebuild the initial ramdisk image"

# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak
# dracut -f -v

Learn More
For additional documentation on LVM filters, see the following

Knowledgebase:
* How do I debug an LVM problem?"
* "What is LVM's filter setting and how do I configure it easily in RHEL?""
* "How do I setup multipath on a system that already has LVM configured?""
* "Cannot activate LVs in VG while PVs appear on duplicate devices"
* "LVM commands print "WARNING: Duplicate VG name " in RHEL"
* "WARNING: Duplicate VG name"
* "LVM reports "Cannot use device with duplicates." "
* "lvm commands are not finding or displaying the expected volumes in RHEL"
* "Some LVM commands falsely report missing PVs even though the corresponding disks are present, have correct metadata and they are not marked with the "MISSING" flag."
* "WARNING: PV /dev/sdX in VG vgXX is using an old PV header, modify the VG to update"

Documentation:
RHEL 8 - Configuring and managing logical volumes
RHEL 7 - Logical Volume Manager Administration
RHEL 6 - Logical Volume Manager Administration
RHEL 5 - Logical Volume Manager Administration