Chapter 6. Troubleshooting LVM
You can use LVM tools to troubleshoot a variety of issues in LVM volumes and groups.
6.1. Gathering diagnostic data on LVM
If an LVM command is not working as expected, you can gather diagnostics in the following ways.
Use the following methods to gather different kinds of diagnostic data:
-vargument to any LVM command to increase the verbosity level of the command output. Verbosity can be further increased by adding additional
v’s. A maximum of four such
v’sis allowed, for example,
logsection of the
/etc/lvm/lvm.confconfiguration file, increase the value of the
leveloption. This causes LVM to provide more details in the system log.
If the problem is related to the logical volume activation, enable LVM to log messages during the activation:
activation = 1option in the
logsection of the
Execute the LVM command with the
- Examine the command output.
If you do not reset the option to
0, the system might become unresponsive during low memory situations.
- Set the
Display an information dump for diagnostic purposes:
Display additional system information:
# lvs -v
# pvs --all
# dmsetup info --columns
Examine the last backup of the LVM metadata in the
/etc/lvm/backup/directory and archived versions in the
Check the current configuration information:
/run/lvm/hintscache file for a record of which devices have physical volumes on them.
- Add the
6.2. Displaying information on failed LVM devices
You can display information about a failed LVM volume that can help you determine why the volume failed.
Display the failed volumes using the
Example 6.1. Failed volume groups
In this example, one of the devices that made up the volume group myvg failed. The volume group is unusable but you can see information about the failed device.
# vgs --options +devices /dev/vdb1: open failed: No such device or address /dev/vdb1: open failed: No such device or address WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s. WARNING: VG myvg is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1). WARNING: Couldn't find all devices for LV myvg/mylv while checking used and assumed devices. VG #PV #LV #SN Attr VSize VFree Devices myvg 2 2 0 wz-pn- <3.64t <3.60t [unknown](0) myvg 2 2 0 wz-pn- <3.64t <3.60t [unknown](5120),/dev/vdb1(0)
Example 6.2. Failed logical volume
In this example, one of the devices failed due to which the logical volume in the volume group failed. The command output shows the failed logical volumes.
# lvs --all --options +devices /dev/vdb1: open failed: No such device or address /dev/vdb1: open failed: No such device or address WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s. WARNING: VG myvg is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/sdb1). WARNING: Couldn't find all devices for LV myvg/mylv while checking used and assumed devices. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices mylv myvg -wi-a---p- 20.00g [unknown](0) [unknown](5120),/dev/sdc1(0)
6.3. Removing lost LVM physical volumes from a volume group
If a physical volume fails, you can activate the remaining physical volumes in the volume group and remove all the logical volumes that used that physical volume from the volume group.
Activate the remaining physical volumes in the volume group:
# vgchange --activate y --partial myvg
Check which logical volumes will be removed:
# vgreduce --removemissing --test myvg
Remove all the logical volumes that used the lost physical volume from the volume group:
# vgreduce --removemissing --force myvg
Optional: If you accidentally removed logical volumes that you wanted to keep, you can reverse the
# vgcfgrestore myvgWarning
If you remove a thin pool, LVM cannot reverse the operation.
6.4. Finding the metadata of a missing LVM physical volume
If the volume group’s metadata area of a physical volume is accidentally overwritten or otherwise destroyed, you get an error message indicating that the metadata area is incorrect, or that the system was unable to find a physical volume with a particular UUID.
This procedure finds the latest archived metadata of a physical volume that is missing or corrupted.
Find the archived metadata file of the volume group that contains the physical volume. The archived metadata files are located at the
# cat /etc/lvm/archive/myvg_00000-1248998876.vg
Replace 00000-1248998876 with the backup-number. Select the last known valid metadata file, which has the highest number for the volume group.
Find the UUID of the physical volume. Use one of the following methods.
List the logical volumes:
# lvs --all --options +devices Couldn't find device with uuid 'FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk'.
Examine the archived metadata file. Find the UUID as the value labeled
id =in the
physical_volumessection of the volume group configuration.
Deactivate the volume group using the
# vgchange --activate n --partial myvg PARTIAL MODE. Incomplete logical volumes will be processed. WARNING: Couldn't find device with uuid 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s. WARNING: VG myvg is missing PV 42B7bu-YCMp-CEVD-CmKH-2rk6-fiO9-z1lf4s (last written to /dev/vdb1). 0 logical volume(s) in volume group "myvg" now active
6.5. Restoring metadata on an LVM physical volume
This procedure restores metadata on a physical volume that is either corrupted or replaced with a new device. You might be able to recover the data from the physical volume by rewriting the metadata area on the physical volume.
Do not attempt this procedure on a working LVM logical volume. You will lose your data if you specify the incorrect UUID.
- You have identified the metadata of the missing physical volume. For details, see Finding the metadata of a missing LVM physical volume.
Restore the metadata on the physical volume:
# pvcreate --uuid physical-volume-uuid \ --restorefile /etc/lvm/archive/volume-group-name_backup-number.vg \ block-deviceNote
The command overwrites only the LVM metadata areas and does not affect the existing data areas.
Example 6.3. Restoring a physical volume on /dev/vdb1
The following example labels the
/dev/vdb1device as a physical volume with the following properties:
The UUID of
The metadata information contained in
VG_00050.vg, which is the most recent good archived metadata for the volume group
# pvcreate --uuid "FmGRh3-zhok-iVI8-7qTD-S5BI-MAEN-NYM5Sk" \ --restorefile /etc/lvm/archive/VG_00050.vg \ /dev/vdb1 ... Physical volume "/dev/vdb1" successfully created
- The UUID of
Restore the metadata of the volume group:
# vgcfgrestore myvg Restored volume group myvg
Display the logical volumes on the volume group:
# lvs --all --options +devices myvg
The logical volumes are currently inactive. For example:
LV VG Attr LSize Origin Snap% Move Log Copy% Devices mylv myvg -wi--- 300.00G /dev/vdb1 (0),/dev/vdb1(0) mylv myvg -wi--- 300.00G /dev/vdb1 (34728),/dev/vdb1(0)
If the segment type of the logical volumes is RAID, resynchronize the logical volumes:
# lvchange --resync myvg/mylv
Activate the logical volumes:
# lvchange --activate y myvg/mylv
If the on-disk LVM metadata takes at least as much space as what overrode it, this procedure can recover the physical volume. If what overrode the metadata went past the metadata area, the data on the volume may have been affected. You might be able to use the
fsckcommand to recover that data.
Display the active logical volumes:
# lvs --all --options +devices LV VG Attr LSize Origin Snap% Move Log Copy% Devices mylv myvg -wi--- 300.00G /dev/vdb1 (0),/dev/vdb1(0) mylv myvg -wi--- 300.00G /dev/vdb1 (34728),/dev/vdb1(0)
6.6. Rounding errors in LVM output
LVM commands that report the space usage in volume groups round the reported number to
2 decimal places to provide human-readable output. This includes the
As a result of the rounding, the reported value of free space might be larger than what the physical extents on the volume group provide. If you attempt to create a logical volume the size of the reported free space, you might get the following error:
Insufficient free extents
To work around the error, you must examine the number of free physical extents on the volume group, which is the accurate value of free space. You can then use the number of extents to create the logical volume successfully.
6.7. Preventing the rounding error when creating an LVM volume
When creating an LVM logical volume, you can specify the size of the logical volume to avoid rounding error.
Find the number of free physical extents in the volume group:
# vgdisplay myvg
Example 6.4. Free extents in a volume group
For example, the following volume group has 8780 free physical extents:
--- Volume group --- VG Name myvg System ID Format lvm2 Metadata Areas 4 Metadata Sequence No 6 VG Access read/write [...] Free PE / Size 8780 / 34.30 GB
Create the logical volume. Enter the volume size in extents rather than bytes.
Example 6.5. Creating a logical volume by specifying the number of extents
# lvcreate --extents 8780 --name mylv myvg
Example 6.6. Creating a logical volume to occupy all the remaining space
Alternatively, you can extend the logical volume to use a percentage of the remaining free space in the volume group. For example:
# lvcreate --extents 100%FREE --name mylv myvg
Check the number of extents that the volume group now uses:
# vgs --options +vg_free_count,vg_extent_count VG #PV #LV #SN Attr VSize VFree Free #Ext myvg 2 1 0 wz--n- 34.30G 0 0 8780