RH linux 5 server does not boot anymore after kernel update

Latest response

Hi,

 

I have some Red Hat 5 Linux servers running kernel version: 2.6.18-164.el5. I need to upgrade at least to version 2.6.18-185.el5 because I want to extend a LVM ext3-partition with more Gbytes. I have downloaded version 1.6.18-255.el5 and did the upgrade on a virtual testserver. On a Xenserver platform it worked but on a Hyper-V server as host, the virtual Linux machine does not boot anymore.

 

# rpm -U `kernel-files`    (was used as upgrade command)

 

This seems to work but when i reboot the server it does not startup anymore. It fails on finding VolGroup00.

 

After putting back the .VHD of the virtual machine, i could get my server back up with the old kernel.

 

Is this a known issue ? Is there another kernel more recent than 2.6.18-185 that could help ?

 

Rgds,

 

 

 

IO
Log in to join the conversation

Responses

John Ruemker's picture Red Hat Guru 4815 points
26 April 2011 1:43 PM

Hello,

I am unclear as to why you need to update your kernel in order to resize your LVM2 logical volume and corresponding ext3 file system.  All available RHEL 5 kernels, including 2.6.18-164.el5, are able to resize logical volumes.   Could you clarify this please?

 

Regarding the system not finding VolGroup00: did you make any changes to your LVM layout before you rebooted?  For instance, did you extend the volume group and logical volume?  Did you make any changes to /boot/grub/grub.conf or /etc/lvm/lvm.conf?  If you could post the full set of steps that you followed, it may help us understand what went wrong.

 

Can you also check your /etc/lvm/lvm.conf for the line 'filter = [ ... ]' and post it here?

 

Thanks,

John Ruemker, RHCA
Red Hat Technical Account Manager

Sadique Puthen's picture Red Hat Guru 4649 points
26 April 2011 2:31 PM

Was this linux guest running with "Linux Integration Services" installed or without it on the old kernel?

IO Community Member 96 points
26 April 2011 2:40 PM

Hi,

 

Thanks for your quick reply.

 

1. This operation is the result of reading of the following article:

https://bugzilla.redhat.com/show_bug.cgi?id=525100

 

If you read the whole ticket you see at the end the DEVs reply it's a bug and is fixed in kernel 2.6.18-185.el5 and s-olaf-vprod237 uses kernel 2.6.18-164.el5 , so the bug still occurs in this. (Due to a lock congestion)

If we ever want to extend filesystems online (Which is kind of the idea with LVM) we will need to upgrade our kernel to this version in the near future.

 

Or is this the wrong conclusion ? I took kernel version V 2.6.18-255.el5 which is more recent than 2.6.18-185

 

2. I did the following

 

# rpm -qa | grep kernel    (to see the installed packages)

 

- download the version V 2.6.18-255.el5

 

# rpm -i --test     (of the concerned .rpms) failed with conflicts

 

# rpm -U --test     (of the concerned .rpms) OK

 

# rpm -U *           (of the concerned .rpms) OK

 

The installation went well but the server did not boot with the messages specified below:

 

Reading all physical volumes. This may take a while....
Volume Group "VolGroup00" not found
unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem '/dev/root´
setuproot :moving /dev failed: no such file or directory
setuproot :error mounting /proc : no such file or directory
setuproot :error mounting /sys : no such file or directory
switchroot: mount failed: no such file or directory

kernel panic - not syncing: Attempted to kill init !

 

Rgds,

 

Johan

 

John Ruemker's picture Red Hat Guru 4815 points
26 April 2011 5:36 PM

Thank you for clearing that up.  You are correct that it is best to be on the later kernel version before performing the resize2fs, to be sure you don't run into any issues such as what's described in the bugzilla.

 

Often times system administrators will reconfigure the lvm filter while the system is up and running, and it won't be discovered right away that this filter change has caused a problem since the volume group is already active.  However when you install a new kernel, it will remake the initrd, which in turn pulls in the latest copy of lvm.conf.  If you had an invalid filter in there that would prevent init from finding the physical volume(s) for VolGroup00, then you would see the issue you have described.

 

Can you post the 'filter' line from your /etc/lvm/lvm.conf?

 

Also, can you let us know if booting from the old kernel entry in grub allows the system to boot normally?

 

Thanks,
John Ruemker, RHCA

Red Hat Technical Account Manager

IO Community Member 96 points
27 April 2011 6:52 AM

Hi John,

 

1. Please find below the filter lines of /etc/lvm/lvm.conf

 

    # By default we accept every block device:
    filter = [ "a/.*/" ]

    # Exclude the cdrom drive
    # filter = [ "r|/dev/cdrom|" ]

    # When testing I like to work with just loopback devices:
    # filter = [ "a/loop/", "r/.*/" ]

    # Or maybe all loops and ide drives except hdc:
    # filter =[ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.*|" ]

    # Use anchors if you want to be really specific
    # filter = [ "a|^/dev/hda8$|", "r/.*/" ]

    # The results of the filtering are cached on disk to avoid
    # rescanning dud devices (which can take a very long time).
    # By default this cache is stored in the /etc/lvm/cache directory
    # in a file called '.cache'.
    # It is safe to delete the contents: the tools regenerate it.
    # (The old setting 'cache' is still respected if neither of
    # these new ones is present.)
    cache_dir = "/etc/lvm/cache"
    cache_file_prefix = ""

 

2. As rpm -i gave a conflict and so failed and rpm -U removes the old kernel, I had to restore the .VHD corresponding with the old kernel to be able to boot (with the old kernel)

 

Rgds,

 

Johan

 

 

 

 

John Ruemker's picture Red Hat Guru 4815 points
27 April 2011 1:54 PM

Thank you for providing that data.  Your filter line isn't to blame then, since you are using the default which causes LVM to scan all devices for physical volumes.

 

Unfortunately I can't explain why you would be seeing this problem, using the data we have.  With a look at your initrd's contents, we might be able to tell more.  Would you be able to open a case with Red Hat Global Support Services so that we can collect some data from your system and do a bit more digging?  Let me know if you need any information on how to open a case. 

 

Regards,

John Ruemker, RHCA
Red Hat Technical Account Manager

IO Community Member 96 points
28 April 2011 9:16 AM

John,

 

I will open a case today at RH.

 

Best regards,

 

Johan

ir. Jan Gerrit Kootstra's picture Guru 6222 points
28 April 2011 12:11 PM

Johan,

 

In general I would never upgrade a kernel.

 

I would use up2date or yum for these tools know whether a new package should be installed or updated.

 

The kernel itself I will install, so that one is able to try to reboot to the old version.

 

 

Kind regards,

 

 

ir. Jan Gerrit Kootstra

Sadique Puthen's picture Red Hat Guru 4649 points
28 April 2011 12:17 PM

Wish to repeat this once again. You might have missed my earlier comment on this.

 

Was this linux guest running with "Linux Integration Services" installed or without it on the old kernel?

JR Community Member 20 points
7 May 2011 12:43 AM

I am having the same problem as in this ticket...I actually just opened a ticket for this in Redhat support but then found this thread. All I did was install the base Linux OS...then added it as one my supported servers and then ran "yum update". There were about 600 updates in this and one of them was the kernel.

 

After updates were completed and I restarted the server I ran into this issue. I can go to GRUB during inital startup and slect the old kernel version and I am able to boot the system just fine.

 

I have not made any changes to anything on the server. I did fresht install on our Hyper-V server (i.e., VM). the lvm.conf file is the default one that was there and no changes have been done.

 

Any insight on this would be greatly appreciated.

Jack Waterworth's picture Red Hat Expert 852 points
14 May 2011 1:38 AM

I am also hitting this problem but I am not aware of any recent updates I made to my machine. I am currently going through the machine and will let you know if I find a solution. I may have made some configuration changes between reboots.  The error seems to indicate that the volume group that is specified in my initrd for root is not detected. 

Phil Jensen's picture Guru 3262 points
19 May 2011 2:38 AM

Sadique mentioned this earlier, but I want to touch on it too.

 

It is possible that you need to rebuild your init ramdisk (initrd) with support for the Microsoft paravirtualized block driver.  A new initrd is generated and used with your newly installed kernel.

 

Please refer to your Linux Integration Services documentation for additional details.

 

Phil Jensen, RHCA

Sr. Operations Engineer

Proofpoint, Inc.

Jack Waterworth's picture Red Hat Expert 852 points
19 May 2011 6:34 PM

My issue appears to be corrupted LVM metadata on my primary PV. I am working to restore it.

MW Newbie 15 points
24 February 2012 10:01 AM

Hi,  I've had similar issues with RHEL 5 servers not rebooting after patching to the latest kernel.  The resolution in our case was to remove 'noapic' from the kernel boot parameters in grub.conf.

 

MK Community Member 25 points
11 February 2013 8:15 PM

Hi, I'm running RHEL 5.5 over 3 years without Linux Integration Services on my Hyper-V cluster and could apply all kernel updates without any problems until the last update 2.6.18-348.1.1 - atfer this update the OS does not boot anymore. My last functional kernel was 2.6.18-308.16.1 - it worked without any problems. With the latest kernel I get the same message -  Volume group "VolGroup00" not found - this results in kernel panic. I'm able to boot with the previous kernel 2.6.18-308.16.1 - the kernel boot parameters are for both kernel versions identical.

 

 

 

 

John Ruemker's picture Red Hat Guru 4815 points
11 February 2013 8:29 PM

Hello,

In many of these situations where a newer kernel fails to boot where the old one still does, its not so much the kernel thats at fault, its the new initrd that was built for that kernel.  If you have changed anything in your /etc/lvm/lvm.conf (such as the filter) since the last kernel was installed, your system may have continued to boot fine until now but once that lvm.conf is included in an initrd, your boot process and devices will depend on the values in that file; if there were a syntax error or a setting which would cause your root Volume Group to not be found, the act of installing a new kernel would include it in the initrd and booting off that new kernel would panic.  Similar to how if you were booting from a multipath device and you had modified multipath.conf, it may not have had a noticeable impact until you were actually mapping the boot device in the initrd using that new configuration.  Or if you had modified the options for the SCSI controller's kernel module in /etc/modprobe.conf, it wouldn't have actually been applied until those options were included in the initrd.

So, the best way to solve these types of problems is to actually inspect the contents of the initrd to see what changed.  This can be a complicated process, so Red Hat Global Support Services can certainly help you with it if you wish to open a case.  But if you'd rather try it on your own, you can unpack each of the initrd's (in RHEL 5) like so:

# mkdir initrd-old

# cd initrd-old

# zcat /boot/initrd-2.6.18-Y1.Z1.el5.img | cpio -idv

# mkdir ../initrd-new

# cd ../initrd-new

# zcat /boot/initrd-2.6.18-Y2.Z2.el5.img | cpio -idv

(filling in the correct filename for the old and new initrds, respectively).  Now that you have them unpacked, you can compare the files with diff or however is convenient for you.  If you can spot a few differences, its usually not too difficult to figure out which is to blame.  If you start in the files mentioned above (lvm.conf, multipath.conf, modprobe.conf, etc), those are usually the common culprits in these scenarios.

Again, don't hesitate to open a case if you'd like some help with this.

Regards,

John Ruemker, RHCA

Senior Software Maintenance Engineer

Global Support Services

Red Hat, Inc.

Th Guru 2987 points
12 February 2013 6:38 PM

The joys of virtualization environments - particularly if running the virtualization environment's tools within your VMs - is that, if someone updates your VM management tools, it can klotz your modprobe.conf. Had that happen recently and it took doing the above to notice, "oh, crap, the VM management tools ate my disk drivers' entries".

I've also run  into this with systems where I was simulating a physical server by setting up bonded interfaces within the VM. Dunno about Hyper-V's management tools, but VMware's management tools tend to assume a *very* simple configuration and, if you do the automated install, will tend to punish you if you've done anything marginally "complicated" (VMware tools seem to pretty much want to completely own the modprobe.conf files).

james.radtke@siriusxm.com's picture Guru 6863 points
11 February 2013 8:23 PM

I've been running into issues with LVM becoming invalid (for a number of different reasons).  More often in my Virtualized environment.

Assuming you have a method to boot from media (network, CD, etc...) I would recommend booting off that media and then running

vgscan && pvscan && lvscan

vgs && pvs && lvs

then check out the timestamp on your /etc/lvm/lvm.conf

If you use Satellite for deployments you can find a backup copy of the lvm.conf in /var/lib/rhncfg/backups/

If not, I would recommend comparing it to a working system.

rh Red Hat Pro 686 points
12 February 2013 7:51 AM

Hello all,

There is a known issue with mkinitrd while updating to RHEL5.9 kernel. Created an article.

https://access.redhat.com/knowledge/solutions/298243

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.