Chapter 17. Troubleshooting LVM

Procedure

• Use the following methods to gather different kinds of diagnostic data:

  • Add the -v argument to any LVM command to increase the verbosity level of the command output. Verbosity can be further increased by adding additional v’s. A maximum of four such v’s is allowed, for example, -vvvv.
  • In the log section of the /etc/lvm/lvm.conf configuration file, increase the value of the level option. This causes LVM to provide more details in the system log.

  • If the problem is related to the logical volume activation, enable LVM to log messages during the activation:

    1. Set the activation = 1 option in the log section of the /etc/lvm/lvm.conf configuration file.
    2. Execute the LVM command with the -vvvv option.
    3. Examine the command output.

    4. Reset the activation option to 0.

       If you do not reset the option to 0, the system might become unresponsive during low memory situations.

  • Display an information dump for diagnostic purposes:

    # lvmdump

  • Display additional system information:

    # lvs -v

    # pvs --all

    # dmsetup info --columns

  • Examine the last backup of the LVM metadata in the /etc/lvm/backup/ directory and archived versions in the /etc/lvm/archive/ directory.

  • Check the current configuration information:

    # lvmconfig

  • Check the /run/lvm/hints cache file for a record of which devices have physical volumes on them.

Prerequisites

• You have identified the metadata of the missing physical volume. For details, see Finding the metadata of a missing LVM physical volume.

Verification steps

• Display the active logical volumes:

  # lvs --all --options +devices
  
    LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
   mylv myvg   -wi--- 300.00G                               /dev/vdb1 (0),/dev/vdb1(0)
   mylv myvg   -wi--- 300.00G                               /dev/vdb1 (34728),/dev/vdb1(0)

Verification steps

• Check the number of extents that the volume group now uses:

  # vgs --options +vg_free_count,vg_extent_count
  
    VG     #PV #LV #SN  Attr   VSize   VFree  Free  #Ext
    myvg   2   1   0   wz--n- 34.30G    0    0     8780

• Is found in label_header and pv_header structures.
• Is in the second 512-byte sector of the disk. Note that if a non-default location was specified when creating the physical volume (PV), the header can also be in the first or third sector.

• Begins 4096 bytes from the start of the disk.
• Ends 1 MiB from the start of the disk.
• Begins with a 512 byte sector containing the mda_header structure.

Procedure

• Print current metadata text as referenced from valid mda_header:

  # pvck --dump metadata <disk>

  Example 17.8. Metadata text from valid mda_header

  # pvck --dump metadata /dev/sdb
    metadata text at 172032 crc Oxc627522f # vgname test segno 59
    ---
    <raw metadata from disk>
    ---

• Print the locations of all metadata copies found in the metadata area, based on finding a valid mda_header:

  # pvck --dump metadata_all <disk>

  Example 17.9. Locations of metadata copies in the metadata area

  # pvck --dump metadata_all /dev/sdb
    metadata at 4608 length 815 crc 29fcd7ab vg test seqno 1 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 7168 length 1450 crc 5652ea55 vg test seqno 3 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv

• Search for all copies of metadata in the metadata area without using an mda_header, for example, if headers are missing or damaged:

  # pvck --dump metadata_search <disk>

  Example 17.10. Copies of metadata in the metadata area without using an mda_header

  # pvck --dump metadata_search /dev/sdb
    Searching for metadata at offset 4096 size 1044480
    metadata at 4608 length 815 crc 29fcd7ab vg test seqno 1 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 7168 length 1450 crc 5652ea55 vg test seqno 3 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv

• Include the -v option in the dump command to show the description from each copy of metadata:

  # pvck --dump metadata -v <disk>

  Example 17.11. Showing description from each copy of metadata

  # pvck --dump metadata -v /dev/sdb
    metadata text at 199680 crc 0x628cf243 # vgname my_vg seqno 40
    ---
  my_vg {
  id = "dmEbPi-gsgx-VbvS-Uaia-HczM-iu32-Rb7iOf"
  seqno = 40
  format = "lvm2"
  status = ["RESIZEABLE", "READ", "WRITE"]
  flags = []
  extent_size = 8192
  max_lv = 0
  max_pv = 0
  metadata_copies = 0
  
  physical_volumes {
  
  pv0 {
  id = "8gn0is-Hj8p-njgs-NM19-wuL9-mcB3-kUDiOQ"
  device = "/dev/sda"
  
  device_id_type = "sys_wwid"
  device_id = "naa.6001405e635dbaab125476d88030a196"
  status = ["ALLOCATABLE"]
  flags = []
  dev_size = 125829120
  pe_start = 8192
  pe_count = 15359
  }
  
  pv1 {
  id = "E9qChJ-5ElL-HVEp-rc7d-U5Fg-fHxL-2QLyID"
  device = "/dev/sdb"
  
  device_id_type = "sys_wwid"
  device_id = "naa.6001405f3f9396fddcd4012a50029a90"
  status = ["ALLOCATABLE"]
  flags = []
  dev_size = 125829120
  pe_start = 8192
  pe_count = 15359
  }

Procedure

• If -f <filename> is added to --dump metadata, the raw metadata is written to the named file. You can use this file for repair.
• If -f <filename> is added to --dump metadata_all or --dump metadata_search, then raw metadata from all locations is written to the named file.

• To save one instance of metadata text from --dump metadata_all|metadata_search add --settings "metadata_offset=<offset>" where <offset> is from the listing output "metadata at <offset>".

  Example 17.12. Output of the command

  # pvck --dump metadata_search --settings metadata_offset=5632 -f meta.txt /dev/sdb
    Searching for metadata at offset 4096 size 1044480
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
  # head -2 meta.txt
  test {
  id = "FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv"

Prerequisites

• You have identified the metadata of the missing physical volume. For details, see Finding the metadata of a missing LVM physical volume.

Verification

• To verify that the new physical volume is intact and the volume group is functioning correctly, check the output of the following command:

Procedure

• Repair the headers and metadata on the disk:

  # pvck --repair -f <metadata-file> <disk>

  where

  • <metadata-file> is a file containing the most recent metadata for the VG. This can be /etc/lvm/backup/vg-name, or it can be a file containing raw metadata text from the pvck --dump metadata_search command output.
  • <disk> is the name of the disk where the PV is supposed to be, for example, /dev/sdb. To prevent data loss, verify that is the correct disk. If you are not certain the disk is correct, contact Red Hat Support.

Verification

• To check that the new physical volume is intact and the volume group is functioning correctly, check outputs of the following commands:

  # vgs <vgname>

  # pvs <pvname>

  # lvs <lvname>