Unable to boot my Red Hat Enterprise Linux 5 system when it detects a Duplicate Volume Group using external storage

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • HP Service Guard Cluster management software
    • Shared volume groups
  • Sybase database software

Issue

  • System unable to boot after disaster recovery testing

  • Going into recovery shell because it was failing fsck on a device

An error occurred during the file system check.
Dropping you to shell; the system will reboot when you leave the shell.
Give root password for maintenance.

bash-3.2$

  • Error when booting about duplicate volume group.

Reading all physical volumes.  This may take a while...
WARNING: Duplicate VG name VolGroup01: Existing 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
  Found volume group "VolGroup01" using metadata type lvm2
  Found volume group "VolGroup01" using metadata type lvm2

Resolution

Root Cause

  • It appears that a LUN that should only be presented to 1 cluster node was inadvertently presented to both nodes in the cluster.  This was causing a volume group name conflict because there was already a disk with that volume group name on that host.

Diagnostic Steps

  • Check the output of vgdisplay -vvv to see if there are any duplicate volume group entries:
    Node 1:

    $ vgdisplay -vvv | grep WARNING         
      WARNING: Duplicate VG name VolGroup01: Existing 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: Existing 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: Existing 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      WARNING: Duplicate VG name VolGroup01: 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ (created here) takes precedence over UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2Node 2:

$ grep WARNING vgdisplay  
<no output>

  • Check which disk is the one that is mistakenly presented to both nodes:
    Node 1:

    $ vgdisplay -vvv | grep 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ
      VG UUID               4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ
    $ vgdisplay -vvv | grep UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
      VG UUID               UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2Node 2:

$ vgdisplay -vvv | grep 4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ
<no output>
$ vgdisplay -vvv | grep UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2
  VG UUID               UHG9ph-suCH-2V8B-wM0F-7DhM-I2Yd-hR8fB2

  • Find out which device it is by looking through the output of vgdisplay -v:

    $ vgdisplay -v
    <some data omitted>
    --- Volume group ---
      VG Name               vgsybtemp
      System ID            
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  6
      VG Access             read/write
      VG Status             exported/resizable
      MAX LV                0
      Cur LV                2
      Open LV               0
      Max PV                64
      Cur PV                1
      Act PV                1
      VG Size               67.42 GB
      PE Size               8.00 MB
      Total PE              8630
      Alloc PE / Size       7680 / 60.00 GB
      Free  PE / Size       950 / 7.42 GB
      VG UUID               4QALCw-wBOD-joji-2Qzf-GFLz-2uk0-iK2cwJ
      
      --- Physical volumes ---
      PV Name               /dev/dm-53    
      PV UUID               7qSez4-2DyH-hHYn-anwq-mcaM-s1Vm-hjOMiI
      PV Status             allocatable
      Total PE / Free PE    8630 / 950

  • Device /dev/dm-53 is the device that needs to be removed from the system.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments