Kernel panic on boot following "dracut Warning: LVM rootvg/rootlv not found"

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7

Issue

  • On boot, the system has a kernel panic. Prior to the panic, we see a dracut warning similar to the following, indicating that the root VG and LV are not found:
dracut Warning: LVM rootvg/rootlv not found

Resolution

To resolve this issue, one or both of the following may be required, followed by rebuilding the initial ramdisk:

  • Repair the LVM filter in /etc/lvm/lvm.conf to ensure it accepts the device associated with the root filesystem.
  • Ensure that the root VG and LV paths references in the GRUB configuration are correct.
  • Ensure LVM package is installed and lvm module is present in initramfs image file.

The following steps offer one approach to troubleshooting and resolving this, making use of the dracut shell:

Enter the dracut shell

The dracut shell will replace a kernel panic after an error in the initial ramdisk, allowing you to troubleshoot and possibly resolve the issue from there. To enter the dracut shell in place of the panic, interrupt the GRUB boot process, edit the kernel command line as follows, and then continue the boot process:

- Remove these (if present): `rhgb quiet`
- Add this: `rdshell`

Continue to boot with this change. After the dracut warning, you should see something like this:

Dropping to debug shell.

sh: can't access tty; job control turned off
dracut:/#

View and repair the LVM filter in /etc/lvm/lvm.conf

1. From the dracut shell described in the first section, run the following commands at the prompt:

dracut:/# lvm pvs
dracut:/# lvm lvs

If the root VG and LVs are shown in the output, skip to the next section on repairing the GRUB configuration. If they are missing, go on to the next step.

2. Remove the LVM configuration file in the currently running initial ramdisk. Note that this does not persist, and it does not affect the root filesystem. It is only a change in what is currently loaded in RAM for the ramdisk. This will force LVM to use its default "accept all" filter when scanning for devices:

dracut:/# rm /etc/lvm/lvm.conf

3. Re-scan devices with this LVM filter removed and confirm that the root VG was found:

dracut:/# lvm pvscan
dracut:/# lvm vgscan
dracut:/# lvm lvscan

4. If the root VG was found, skip this step. If the root VG was not found, check the output of blkid to confirm whether the physical block device was found. If the devices itself is not showing, examine physical connections to the device. If the device is showing, investigate whether there was corruption on the device causing LVM data to be unreadable. Reach our to Red Hat Support for assistance as needed:

dracut:/# blkid   
/dev/vda1: UUID="6469e70f-938b-4046-aefe-7a0c1ad1809b" TYPE="ext4" 
/dev/vda2: UUID="L4P8yb-1dCG-H4Je-uvpj-pNuK-VBRr-eTkmsG" TYPE="LVM2_member"

5. If the VG was found in the scans when the lvm.conf was deleted, boot to the rescue environment or to an alternate kernel (if possible) and adjust the "filter =" line in /etc/lvm/lvm.conf as appropriate. For fastest recovery, switch to the default filter so the system can boot normally. You can revisit the filter later to customize it as needed. The default "accept all" filter is shown here:

filter = [ "a/.*/" ]

6. Rebuild the initial ramdisk following changes to /etc/lvm/lvm.conf, prior to rebooting the system:
How to rebuild the initial ramdisk image in Red Hat Enterprise Linux

7. Try rebooting the system normally. If the same error occurs again, even with a corrected filter, see the next section covering the GRUB configuration.

View and repair the GRUB configuration

1. From the dracut shell described in the first section, run the "lvm pvs" and "lvm lvs" commands at the prompt. Make note of the output, specifically the device, VG, and LV names. The following shows example output with the commands:

dracut:/# lvm pvs
File descriptor 9 (/.console_lock) leaked on lvm invocation. Parent PID 1683: sh
  PV         VG       Fmt  Attr PSize PFree
  /dev/vda2  vg_stark lvm2 a--  9.51g    0 
dracut:/# lvm lvs
File descriptor 9 (/.console_lock) leaked on lvm invocation. Parent PID 1674: sh
  LV           VG             Attr          LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_root   vg_stark -wi------- 8.51g                                                    
  lv_swap vg_stark -wi------- 1.00g

2. Reboot the system and interrupt the GRUB boot loader again. From there, correct the kernel line as needed to ensure that the LV and VG names match with the output shown. Then, continue booting normally.

NOTE: If the system still fails to boot normally with these changes in place, proceed with booting into the rescue environment for further troubleshooting.

3. After booting, edit the GRUB configuration to make the kernel command line changes permanent.

4. Reboot again to confirm that the GRUB changes were successful.

Root Cause

One reason for this message is that the LVM filter, set using the "filter" parameter in /etc/lvm/lvm.conf, is too restrictive. As a result, when the initial ramdisk scans for devices, it rejects the device that contains the root filesystem. Then, when the ramdisk goes on to look for LVM metadata and the root filesystem, it is not found in the remaining devices.

Another reason for this message is that the LVM configuration on the disk is inconsistent with the configuration in the initial ramdisk or in GRUB. An example of this would be renaming the root VG or LV without updating GRUB or rebuilding the ramdisk.

Diagnostic Steps

To determine if the LVM filter may be an issue, run you can take additional steps to investigate from the dracut shell:

1. Interrupt the GRUB boot process and edit the kernel command line as follows:
Remove these (if present): rhgb quiet
Add this: rdshell
The rdshell will drop to the dracut shell if there are problems with the initial ramdisk.

2. Use basic pvs and lvs commands from LVM to check what the current LVM filters have found:

dracut:/# lvm pvs
dracut:/# lvm lvs

If the LVM filter is restrictive and skipping the device with the root filesystem, the output of this command will be missing the root VG and LVs.

3. Check the LVM filter associated with the initial ramdisk (NOTE: this is the file within the ramdisk and not on the root filesystem). The first command shown will let you browse the whole /etc/lvm/lvm.conf file; the second command will only show only lines with "filter" in them.

dracut:/# cat /etc/lvm/lvm.conf | more
dracut:/# cat /etc/lvm/lvm.conf | grep -i filter

The line starting with "filter = " that is not preceded by a comment character ("#") is the active LVM filter. The default value is to accept all devices ("a/.*/").

4. Find the root VG using a PV scan and an overriding LVM filter that accepts all devices. The output may show some errors or warnings, but it should find the root VG in its scan. The left column should show the device path under the PV column and the root VG under the VG column. The following shows the command and an example root VG in the output:

dracut:/# lvm pvs -a --config 'devices { filter = ["a|.*|"]}'
PV                           VG            Fmt    Attr  PSize       PFree
...
/dev/sda3            rootvg      lvm2  a--    125.00g   58.75g

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments