Logical Volume doesn't get activated when a disk containing lvm snapshot in lvm raid1 is missing

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 10
  • LVM RAID 1

Issue

  • Logical Volume does not get activated when a disk containing lvm snapshot in lvm raid1 is missing.

  • Suppose a disk in LVM RAID 1 goes missing or fails and you have to shut down the machine to replace the failed disk. Once the replacement is done, the new disk needs to be added back into the volume group of LVM raid1. When trying to activate the volume group or logical volume, We get the below error:

    [root@rhel9 ~]# lvchange -ay --activationmode degraded vg5/lv5
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: VG vg5 is missing PV aaaa-bbbb-cccc-dddd-eeee-ffff-gggg (last written to [unknown]).
  Aborting.  LV vg5/snaplv5_aug31 is now incomplete and '--activationmode partial' was not specified. <<<

[root@rhel9 ~]# lvs
  Devices file PVID aaaa-bbbb-cccc-dddd-eeee-ffff-gggg  last seen on /dev/sdb not found.
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg .
  WARNING: VG vg5 is missing PV aaaa-bbbb-cccc-dddd-eeee-ffff-gggg  (last written to /dev/sdb).
  LV             VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                    
  lv5            vg5           owi---r-p- 500.00m             <<<                                                
  lv6            vg5           rwi---r-p- 300.00m                                                                                                   
  snaplv5_aug31  vg5           swi---s-p- 152.00m      lv5    <<<

Resolution

  • Remove any existing LVM snapshots that were backed by a failed disk:

    [root@rhel9 ~]# lvremove /dev/vg5/snaplv5_aug31 
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: VG vg5 is missing PV aaaa-bbbb-cccc-dddd-eeee-ffff-gggg (last written to [unknown]).
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  Logical volume "snaplv5_aug31" successfully removed.

  • Once removed, activate the logical volume:

    [root@tpmrhel9 ~]# lvchange -ay --activationmode degraded vg5/lv5
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: VG vg5 is missing PV aaaa-bbbb-cccc-dddd-eeee-ffff-gggg(last written to [unknown]).

[root@tpmrhel9 ~]# lvs
  Devices file PVID aaaa-bbbb-cccc-dddd-eeee-ffff-gggg last seen on /dev/sdb not found.
  WARNING: Couldn't find device with uuid aaaa-bbbb-cccc-dddd-eeee-ffff-gggg.
  WARNING: VG vg5 is missing PV aaaa-bbbb-cccc-dddd-eeee-ffff-gggg (last written to [unknown]).
  LV             VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv5            vg5           rwi-a-r-p- 500.00m                                    100.00          <<<                                                  
  lv6            vg5           rwi-a-r-p- 300.00m                                    100.00

  • Internal Jira has been logged for lvm2 to improve the visibility of those errors at lvm level. This improvement in the error logging has been added in lvm2-2.03.33-2.el9 or higher version. Once upgraded, One can see the warning as below:

    [root@testbox]# vgchange -ay vg1
  WARNING: Couldn't find device with uuid mEvnym-sE8S-7ci4-C2QP-KkCl-6Wtp-S7zSN4.
  WARNING: VG vg1 is missing PV mEvnym-sE8S-7ci4-C2QP-KkCl-6Wtp-S7zSN4 (last written to [unknown]).
  Activating raid LV vg1/lv1 requires the removal of partial snapshot vg1/lvsnap.  <<<
  0 logical volume(s) in volume group "vg1" now active
[root@testbox ~]#

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments