Chapter 18. Troubleshooting LVM

You can use LVM tools to troubleshoot a variety of issues in LVM volumes and groups.

18.1. Gathering diagnostic data on LVM

If an LVM command is not working as expected, you can gather diagnostics in the following ways.

Procedure

  • Use the following methods to gather different kinds of diagnostic data:

    • Add the -vvvv argument to any LVM command to increase the verbosity level of the command output.
    • In the log section of the /etc/lvm/lvm.conf configuration file, increase the value of the level option. This causes LVM to provide more details in the system log.
    • If the problem is related to the logical volume activation, enable LVM to log messages during the activation:

      1. Set the activation = 1 option in the log section of the /etc/lvm/lvm.conf configuration file.
      2. Run the LVM command with the -vvvv option.
      3. Examine the command output.
      4. Reset the activation option to 0.

        If you do not reset the option to 0, the system might become unresponsive during low memory situations.

    • Display an information dump for diagnostic purposes:

      # lvmdump
    • Display additional system information:

      # lvs -v
      # pvs --all
      # dmsetup info --columns
    • Examine the last backup of the LVM metadata in the /etc/lvm/backup/ directory and archived versions in the /etc/lvm/archive/ directory.
    • Check the current configuration information:

      # lvmconfig
    • Check the /run/lvm/hints cache file for a record of which devices have physical volumes on them.

Additional resources

  • The lvmdump(8) man page

18.2. Displaying information on failed LVM devices

You can display information about a failed LVM volume that can help you determine why the volume failed.

Procedure

  • Display the failed volumes using the vgs or lvs utility.

    Example 18.1. Failed volume groups

    In this example, one of the devices that made up the volume group vg failed. The volume group is unusable but you can see information about the failed device.

    # vgs --options +devices
    
      /dev/sdb: open failed: No such device or address
      /dev/sdb: open failed: No such device or address
      WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s.
      WARNING: VG vg is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1).
      WARNING: Couldn't find all devices for LV vg/linear while checking used and assumed devices.
      WARNING: Couldn't find all devices for LV vg/stripe while checking used and assumed devices.
      VG #PV #LV #SN Attr   VSize  VFree  Devices
      vg   2   2   0 wz-pn- <3.64t <3.60t [unknown](0)
      vg   2   2   0 wz-pn- <3.64t <3.60t [unknown](5120),/dev/sdc1(0)

    Example 18.2. Failed linear and striped LV

    In this example, the failed device caused both a linear and a striped logical volume in the volume group to fail. The command output shows the failed logical volumes.

    # lvs --all --options +devices
    
      /dev/sdb: open failed: No such device or address
      /dev/sdb: open failed: No such device or address
      WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s.
      WARNING: VG vg is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1).
      WARNING: Couldn't find all devices for LV vg/linear while checking used and assumed devices.
      WARNING: Couldn't find all devices for LV vg/stripe while checking used and assumed devices.
      LV     VG Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
      linear vg -wi-a---p- 20.00g                                                     [unknown](0)
      stripe vg -wi-a---p- 20.00g                                                     [unknown](5120),/dev/sdc1(0)

    Example 18.3. Failed leg of a mirrored logical volume

    The following examples show the command output from the vgs and lvs utilities when a leg of a mirrored logical volume has failed.

    # vgs --all --options +devices
    
      VG    #PV #LV #SN Attr   VSize VFree Devices
      corey 4 4 0 rz-pnc 1.58T 1.34T my_mirror_mimage_0(0),my_mirror_mimage_1(0)
      corey 4 4 0 rz-pnc 1.58T 1.34T /dev/sdd1(0)
      corey 4 4 0 rz-pnc 1.58T 1.34T unknown device(0)
      corey 4 4 0 rz-pnc 1.58T 1.34T /dev/sdb1(0)
    # lvs --all --options +devices
    
      LV                   VG    Attr   LSize   Origin Snap%  Move Log            Copy%  Devices
      my_mirror corey mwi-a- 120.00G my_mirror_mlog 1.95 my_mirror_mimage_0(0),my_mirror_mimage_1(0)
      [my_mirror_mimage_0] corey iwi-ao 120.00G unknown device(0)
      [my_mirror_mimage_1] corey iwi-ao 120.00G /dev/sdb1(0)
      [my_mirror_mlog] corey lwi-ao 4.00M /dev/sdd1(0)

18.3. Removing lost LVM physical volumes from a volume group

If a physical volume fails, you can activate the remaining physical volumes in the volume group and remove all the logical volumes that used that physical volume from the volume group.

Procedure

  1. Activate the remaining physical volumes in the volume group:

    # vgchange --activate y --partial volume-group
  2. Check which logical volumes will be removed:

    # vgreduce --removemissing --test volume-group
  3. Remove all the logical volumes that used the lost physical volume from the volume group:

    # vgreduce --removemissing --force volume-group
  4. Optional: If you accidentally removed logical volumes that you wanted to keep, you can reverse the vgreduce operation:

    # vgcfgrestore volume-group
    Warning

    If you removed a thin pool, LVM cannot reverse the operation.

18.4. Recovering an LVM physical volume with damaged metadata

If the volume group metadata area of a physical volume is accidentally overwritten or otherwise destroyed, you get an error message indicating that the metadata area is incorrect, or that the system was unable to find a physical volume with a particular UUID. You might be able to recover the data from the physical volume by rewriting the metadata area on the physical volume.

18.4.1. Discovering that an LVM volume has missing or corrupted metadata

The following example shows the command output you might see if the metadata area on a physical volume is missing or corrupted.

Procedure

  • Try to list the logical volumes:

    # lvs --all --options +devices

    Example 18.4. Output with missing or corrupted metadata

    In this example, certain logical volumes are located on a physical volume that has missing or corrupted metadata.

      Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'.
      Couldn't find all physical volumes for volume group VG.
      Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'.
      Couldn't find all physical volumes for volume group VG.
      ...

18.4.2. Finding the metadata of a missing LVM physical volume

This procedure finds the latest archived metadata of a physical volume that is missing or corrupted.

Procedure

  1. Find the archived metadata file of the volume group that contains the physical volume.

    The archived metadata files are located at the /etc/lvm/archive/volume-group-name_backup-number.vg path. Select the last known valid metadata file, which has the highest number for the volume group.

  2. Find the UUID of the physical volume. Use one of the following methods.

    • List the logical volumes:

      # lvs --all --options +devices
      
        Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'.
    • Examine the archived metadata file. Find the UUID as the value labeled id = in the physical_volumes section of the volume group configuration.
    • Deactivate the volume group using the --partial option:

      # vgchange --activate n --partial volume-group-name
      
        PARTIAL MODE. Incomplete logical volumes will be processed.
        WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s.
        WARNING: VG raid_sanity is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1).
        0 logical volume(s) in volume group "raid_sanity" now active

18.4.3. Restoring metadata on an LVM physical volume

This procedure restores metadata on a physical volume that is either corrupted or replaced with a new device.

Warning

Do not attempt this procedure on a working LVM logical volume. You will lose your data if you specify the incorrect UUID.

Prerequisites

Procedure

  1. Restore the metadata on the physical volume:

    # pvcreate --uuid physical-volume-uuid \
               --restorefile /etc/lvm/archive/volume-group-name_backup-number.vg \
               block-device
    Note

    The command overwrites only the LVM metadata areas and does not affect the existing data areas.

    Example 18.5. Restoring a physical volume on /dev/sdh1

    The following example labels the /dev/sdh1 device as a physical volume with the following properties:

    • The UUID of FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk
    • The metadata information contained in VG_00050.vg, which is the most recent good archived metadata for the volume group
    # pvcreate --uuid "FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk" \
               --restorefile /etc/lvm/archive/VG_00050.vg \
               /dev/sdh1
    
      ...
      Physical volume "/dev/sdh1" successfully created
  2. Restore the metadata of the volume group:

    # vgcfgrestore volume-group-name
    
      Restored volume group volume-group-name
  3. Display the logical volumes on the volume group:

    # lvs --all --options +devices volume-group-name

    The logical volumes are currently inactive. For example:

      LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
      stripe VG   -wi--- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
      stripe VG   -wi--- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0)
  4. If the segment type of the logical volumes is RAID or mirror, resynchronize the logical volumes:

    # lvchange --resync volume-group-name/logical-volume-name
  5. Activate the logical volumes:

    # lvchange --activate y /dev/volume-group-name/logical-volume-name
  6. If the on-disk LVM metadata takes at least as much space as what overrode it, this procedure can recover the physical volume. If what overrode the metadata went past the metadata area, the data on the volume may have been affected. You might be able to use the fsck command to recover that data.

Verification steps

  • Display the active logical volumes:

    # lvs --all --options +devices
    
      LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
      stripe VG   -wi-a- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
      stripe VG   -wi-a- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0)

18.5. Replacing a missing LVM physical volume

If a physical volume fails or otherwise needs to be replaced, you can label a new physical volume to replace the one that has been lost in the existing volume group.

Prerequisites

  • You have replaced the physical volume with a new storage device.

    TODO: Reevaluate the placement of this step.

18.5.1. Finding the metadata of a missing LVM physical volume

This procedure finds the latest archived metadata of a physical volume that is missing or corrupted.

Procedure

  1. Find the archived metadata file of the volume group that contains the physical volume.

    The archived metadata files are located at the /etc/lvm/archive/volume-group-name_backup-number.vg path. Select the last known valid metadata file, which has the highest number for the volume group.

  2. Find the UUID of the physical volume. Use one of the following methods.

    • List the logical volumes:

      # lvs --all --options +devices
      
        Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'.
    • Examine the archived metadata file. Find the UUID as the value labeled id = in the physical_volumes section of the volume group configuration.
    • Deactivate the volume group using the --partial option:

      # vgchange --activate n --partial volume-group-name
      
        PARTIAL MODE. Incomplete logical volumes will be processed.
        WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s.
        WARNING: VG raid_sanity is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1).
        0 logical volume(s) in volume group "raid_sanity" now active

18.5.2. Restoring metadata on an LVM physical volume

This procedure restores metadata on a physical volume that is either corrupted or replaced with a new device.

Warning

Do not attempt this procedure on a working LVM logical volume. You will lose your data if you specify the incorrect UUID.

Prerequisites

Procedure

  1. Restore the metadata on the physical volume:

    # pvcreate --uuid physical-volume-uuid \
               --restorefile /etc/lvm/archive/volume-group-name_backup-number.vg \
               block-device
    Note

    The command overwrites only the LVM metadata areas and does not affect the existing data areas.

    Example 18.6. Restoring a physical volume on /dev/sdh1

    The following example labels the /dev/sdh1 device as a physical volume with the following properties:

    • The UUID of FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk
    • The metadata information contained in VG_00050.vg, which is the most recent good archived metadata for the volume group
    # pvcreate --uuid "FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk" \
               --restorefile /etc/lvm/archive/VG_00050.vg \
               /dev/sdh1
    
      ...
      Physical volume "/dev/sdh1" successfully created
  2. Restore the metadata of the volume group:

    # vgcfgrestore volume-group-name
    
      Restored volume group volume-group-name
  3. Display the logical volumes on the volume group:

    # lvs --all --options +devices volume-group-name

    The logical volumes are currently inactive. For example:

      LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
      stripe VG   -wi--- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
      stripe VG   -wi--- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0)
  4. If the segment type of the logical volumes is RAID or mirror, resynchronize the logical volumes:

    # lvchange --resync volume-group-name/logical-volume-name
  5. Activate the logical volumes:

    # lvchange --activate y /dev/volume-group-name/logical-volume-name
  6. If the on-disk LVM metadata takes at least as much space as what overrode it, this procedure can recover the physical volume. If what overrode the metadata went past the metadata area, the data on the volume may have been affected. You might be able to use the fsck command to recover that data.

Verification steps

  • Display the active logical volumes:

    # lvs --all --options +devices
    
      LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
      stripe VG   -wi-a- 300.00G                               /dev/sdh1 (0),/dev/sda1(0)
      stripe VG   -wi-a- 300.00G                               /dev/sdh1 (34728),/dev/sdb1(0)

18.6. Troubleshooting insufficient free extents for a logical volume

You might get the Insufficient free extents error message when attempting to create a logical volume, even when you think that the volume group has enough free space. You can troubleshoot this error to be able to create a logical volume on the volume group.

18.6.1. Volume groups

Physical volumes are combined into volume groups (VGs). This creates a pool of disk space out of which logical volumes can be allocated.

Within a volume group, the disk space available for allocation is divided into units of a fixed-size called extents. An extent is the smallest unit of space that can be allocated. Within a physical volume, extents are referred to as physical extents.

A logical volume is allocated into logical extents of the same size as the physical extents. The extent size is thus the same for all logical volumes in the volume group. The volume group maps the logical extents to physical extents.

18.6.2. Rounding errors in LVM output

LVM commands that report the space usage in volume groups round the reported number to 2 decimal places to provide human-readable output. This includes the vgdisplay and vgs utilities.

As a result of the rounding, the reported value of free space might be larger than what the physical extents on the volume group provide. If you attempt to create a logical volume the size of the reported free space, you might get the following error:

Insufficient free extents

To work around the error, you must examine the number of free physical extents on the volume group, which is the accurate value of free space. You can then use the number of extents to create the logical volume successfully.

18.6.3. Preventing the rounding error when creating an LVM volume

When creating an LVM logical volume, you can specify the size of the logical volume so that no rounding error occurs.

Procedure

  1. Find the number of free physical extents in the volume group:

    # vgdisplay volume-group-name

    Example 18.7. Free extents in a volume group

    For example, the following volume group has 8780 free physical extents:

      --- Volume group ---
      ...
      Free  PE / Size       8780 / 34.30 GB
  2. Create the logical volume. Enter the volume size in extents rather than bytes.

    Example 18.8. Creating a logical volume by specifying the number of extents

    # lvcreate --extents 8780 --name testlv testvg

    Example 18.9. Creating a logical volume to occupy all the remaining space

    Alternately, you can extend the logical volume to use a percentage of the remaining free space in the volume group. For example:

    # lvcreate --extents 100%FREE --name testlv2 testvg

Verification steps

  • Check the number of extents that the volume group now uses:

    # vgs --options +vg_free_count,vg_extent_count
    
      VG     #PV #LV #SN Attr   VSize  VFree Free #Ext
      testvg   2   1   0 wz--n- 34.30G    0     0 8780

18.7. Troubleshooting duplicate physical volume warnings for multipathed LVM devices

When using LVM with multipathed storage, LVM commands that list a volume group or logical volume might display messages such as the following:

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/dm-5 not /dev/sdd
Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/emcpowerb not /dev/sde
Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/sddlmab not /dev/sdf

You can troubleshoot these warnings to understand why LVM displays them, or to hide the warnings.

18.7.1. Root cause of duplicate PV warnings

When a multipath software such as Device Mapper Multipath (DM Multipath), EMC PowerPath, or Hitachi Dynamic Link Manager (HDLM) manages storage devices on the system, each path to a particular logical unit (LUN) is registered as a different SCSI device. The multipath software then creates a new device that maps to those individual paths. Because each LUN has multiple device nodes in the /dev directory that point to the same underlying data, all the device nodes contain the same LVM metadata.

Table 18.1. Example device mappings in different multipath software

Multipath softwareSCSI paths to a LUNMultipath device mapping to paths

DM Multipath

/dev/sdb and /dev/sdc

/dev/mapper/mpath1 or /dev/mapper/mpatha

EMC PowerPath

/dev/emcpowera

HDLM

/dev/sddlmab

As a result of the multiple device nodes, LVM tools find the same metadata multiple times and report them as duplicates.

18.7.2. Cases of duplicate PV warnings

LVM displays the duplicate PV warnings in either of the following cases:

  • The two devices displayed in the output are both single paths to the same device.
  • The two devices displayed in the output are both multipath maps.

Single paths to the same device

The following example shows a duplicate PV warning in which the duplicate devices are both single paths to the same device.

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/sdd not /dev/sdf

If you list the current DM Multipath topology using the multipath -ll command, you can find both /dev/sdd and /dev/sdf under the same multipath map.

These duplicate messages are only warnings and do not mean that the LVM operation has failed. Rather, they are alerting you that LVM uses only one of the devices as a physical volume and ignores the others.

If the messages indicate that LVM chooses the incorrect device or if the warnings are disruptive to users, you can apply a filter. The filter configures LVM to search only the necessary devices for physical volumes, and to leave out any underlying paths to multipath devices. As a result, the warnings no longer appear.

Multipath maps

The following examples show a duplicate PV warning for two devices that are both multipath maps. The duplicate physical volumes are located on two different devices rather than on two different paths to the same device.

Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/mapper/mpatha not /dev/mapper/mpathc
Found duplicate PV GDjTZf7Y03GJHjteqOwrye2dcSCjdaUi: using /dev/emcpowera not /dev/emcpowerh

This situation is more serious than duplicate warnings for devices that are both single paths to the same device. These warnings often mean that the machine is accessing devices that it should not access: for example, LUN clones or mirrors.

Unless you clearly know which devices you should remove from the machine, this situation might be unrecoverable. Red Hat recommends that you contact Red Hat Technical Support to address this issue.

18.7.3. The LVM device filter

LVM tools scan for devices in the /dev directory and check every device there for LVM metadata. A filter in the /etc/lvm/lvm.conf file controls which devices LVM scans.

The filter is a list of patterns that LVM applies to each device found by a scan of the /dev directory, or the directory specified by the dir keyword in the /etc/lvm/lvm.conf file. Patterns are regular expressions delimited by any character and preceded by a for accept or r for reject. The first regular expression in the list that matches a device determines if LVM accepts or rejects (ignores) the device. LVM accepts devices that do not match any patterns.

The following is the default configuration of the filter, which scans all devices:

filter = [ "a/.*/" ]

18.7.4. Example LVM device filters that prevent duplicate PV warnings

The following examples show LVM device filters that avoid the duplicate physical volume warnings that are caused by multiple storage paths to a single logical unit (LUN).

The filter that you configure must include all devices that LVM needs to be check for metadata, such as the local hard drive with the root volume group on it and any multipathed devices. By rejecting the underlying paths to a multipath device (such as /dev/sdb, /dev/sdd, and so on), you can avoid these duplicate PV warnings, because LVM finds each unique metadata area once on the multipath device itself.

  • This filter accepts the second partition on the first hard drive and any DM Multipath devices, but rejects everything else:

    filter = [ "a|/dev/sda2$|", "a|/dev/mapper/mpath.*|", "r|.*|" ]
  • This filter accepts all HP SmartArray controllers and any EMC PowerPath devices:

    filter = [ "a|/dev/cciss/.*|", "a|/dev/emcpower.*|", "r|.*|" ]
  • This filter accepts any partitions on the first IDE drive and any multipath devices:

    filter = [ "a|/dev/hda.*|", "a|/dev/mapper/mpath.*|", "r|.*|" ]

18.7.5. Applying an LVM device filter configuration

This procedure changes the configuration of the LVM device filter, which controls the devices that LVM scans.

Prerequisites

  • Prepare the device filter pattern that you want to use.

Procedure

  1. Test your device filter pattern without modifying the /etc/lvm/lvm.conf file.

    Use an LVM command with the --config 'devices{ filter = [ your device filter pattern ] }' option. For example:

    # lvs --config 'devices{ filter = [ "a|/dev/emcpower.*|", "r|.*|" ] }'
  2. Edit the filter option in the /etc/lvm/lvm.conf configuration file to use your new device filter pattern.
  3. Check that no physical volumes or volume groups that you want to use are missing with the new configuration:

    # pvscan
    # vgscan
  4. Rebuild the initramfs file system so that LVM scans only the necessary devices upon reboot:

    # dracut --force --verbose

18.7.6. Additional resources