YUM Kernel install fails - no depmod, no initramfs, but installs kernel and line in GRUB (w/o initramfs)
- PURPOSE: Looking for troubleshooting assistance, trying to find the root cause
- SITUATION: I've now had a 2nd occurrence of this (among dozens of systems that upgraded just fine), and I cannot figure it out
- PLATFORM Red Hat Enterprise Linux (RHEL) x86-64 Workstation
- RELEASE: Release 6, Update 6 (installed) to Update 8 (upgrade via YUM)
As part of a YUM upgrade of systems from RHEL 6.6 to 6.8, using a local YUM repository (systems do not have Internet access), everything seems to install correctly, then the system is rebooted. It kernel panics, due to no initramfs and modules required to boot.
Upon further inspection, there are no module dependency files in /lib/modules/(version). So the fix is either ...
V=2.6.32-642.1.1.el6.x86_64
depmod -a ${V}
mkinitrd /boot/initramfs-${V}.img ${V}
vi /boot/grub/grub.conf (add initrd line)
Or ...
V=2.6.32-642.1.1.el6.x86_64
rpm -ihv --replacefiles --replacepkgs kernel-${V}.rpm
Either will fix the issue, re-generating the module dependencies and a new initramfs with all required modules and their dependencies.
NOTE: Failure to run depmod first will create an incomplete initramfs (modules with missing module dependencies in the initramfs) that will still kernel panic.
Again, looking for any and all troubleshooting advice, trying to track down the root cause. This only occurs very infrequently, and I cannot find any differences between these systems that failed to have depmod/mkinitrd complete as part of the new kernel install, and those that worked just fine.
Responses
please check above commands
Pro
678 points
I'm not following how this addresses our situation.
I'm also working with Red Hat support, and we are going to break our YUM upgrade commands into separate system v. kernel package updates, with the latter running in debug mode to catch why a post trigger is failing.
Pro
596 points
Are these virtual machines? if yes what hypevisor they are using? we had a similar issue with hyperv instance when there was a need for an iscsi luns.
could you think of a driver that is needed from the initramfs for these specific machines?
Pro
678 points
Bare metal. One of the machines may have had the nVidia kernel module, but the other two did not. But even the nVidia kernel module with DKMS will not 'return' to the prompt (exit) until the module is built.
Also understand in our case, both A) depmod files are not there and B) initramfs doesn't get built at all, period. I'm able to generate an initramfs (not an useful one, but still, at least an initramfs) without depmod files. In our case, neither are generated at all. I've worked in boot-from-SAN and other environments, and am used to a 'missing driver' in the initramfs. In this case, there is not even any initramfs. Hence why I opened the discussion, as well as a 'question' Case.
As I mentioned, per our open Case, we're considering a 2-step, where kernel is updated separate from the rest of the system, and we throw the debug verbosity option.
yum update --rpmverbosity=debug kernel |& tee /tmp/$HOSTNAME.yum.output.log
I believe I had the same issue with the same basic environment. I was using a local YUM repository, and the machine was bare-metal. The kernel was installed and grub.conf was updated, but the initramfs was not getting built. I fixed my case by doing a yum remove of the newest kernel and reinstalling it. I think that is more or less what you've done in your second example using rpm. I don't have a root cause which I'm guessing is what you're looking for though. I just chalked it up to a one-time thing. I updated other clients using the same repository without the same problem.
I did maintenance on about 100 centos 7 hosts yesterday and 5 of them experienced the same issues as reported above. Yum/dracut just 'forgot' to generate a initramfs for the new installed kernel. Grub config was altered with the new kernel but no initramfs line added.
We also have the same issue, need to boot on kernel n-1 version and doing a yum reinstall kernel-3.10.... to have a initramfs...
Pro
678 points
Just an update ...
I'm finding the 'symptoms' include the YUM command failing during the 'cleanup' (after all packages updated/installed), the system kernel panics, and then reboots. This may explain why the kernel is installed and GRUB updated, but neither 'depmod' nor 'initrd' commands run, as I believe it's run via a YUM plug-in during or post 'cleanup'.
This would explain a lot. Something is triggering a kernel panic/reboot during 'cleanup'. It's very difficult to catch, because it happens so quickly. I'm at the point of totally excluding kernel updates as part of YUM. I need to identify what 'cleanup' of a package is causing this too. Maybe GLibC or another, core library?
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.