Why is my server booting with an older kernel despite the default boot kernel being set to the latest version and It being available in the GRUB menu?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9

Issue

  • The default boot kernel is configured to be the latest version, which is also listed in the GRUB menu. However, the server consistently boots with an older kernel by default.

Resolution

  1. Move existing BLS configuration files to a backup directory:

    # mkdir /boot/loader/backup
# mv /boot/loader/entries/*  /boot/loader/backup/

  2. Re-create BLS configuration files:

    # for i in `rpm -q kernel | sed 's/kernel-//g' | sort | xargs echo` ; do kernel-install add $i /lib/modules/$i/vmlinuz ; done

  3. Regenerate the grub configuration file

    • In case of BIOS:
    # grub2-mkconfig -o /boot/grub2/grub.cfg

    • In case of EFI:
      Ref: How to determine if the system is booted in BIOS or UEFI mode

      • In Red Hat Enterprise Linux 7, 8 and 9.0 to 9.2:

        # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

      • In Red Hat Enterprise Linux 9.3 and later:

        # grub2-mkconfig -o /boot/grub2/grub.cfg

        NOTE: The /boot/efi/EFI/redhat/grub.cfg file, which GRUB previously used on UEFI systems, is now a stub that redirects to /boot/grub2/grub.cfg, which contains the real GRUB configuration. In RHEL 9 grub2-mkconfig should not be used with /boot/efi/EFI/redhat/grub.cfg as an output file.

Root Cause

  • The issue is caused by a conflict between the system’s changed machine-id and the GRUB Boot Loader Specification (BLS) sorting logic.

    Identifier Mismatch : The /etc/machine-id file is used as a prefix for the filenames in /boot/loader/entries/. When the machine-id is changed, newly installed kernels use the new ID, while older kernels remain associated with the previous ID.

    Alphanumeric Precedence: The GRUB blscfg module generates the boot menu by performing a reverse-alphanumeric sort on the filenames in /boot/loader/entries/.

    Sort Order Overriding Version: Since the filename begins with the machine-id, GRUB treats the ID as the primary sorting key. If the old machine-id starts with a character that ranks higher alphanumerically (e.g., z... vs. a...) than the new ID, the older kernel entry will always be placed at the top of the list.

    Default Selection: Because GRUB_DEFAULT=0 is configured in /etc/default/grub, the system automatically selects the first entry in that sorted list (Index 0). Consequently, the server boots the older kernel because it sits at the top of the alphanumeric sort, regardless of which kernel version is actually the newest.

Diagnostic Steps

  • Check the default boot kernel:

    # cat /boot/grub2/grubenv           ###### On BIOS server
# cat /boot/efi/EFI/redhat/grubenv  ###### On UEFI server

  • Check if there are BLS configuration files with different machine-id's are present

    # ls -l /boot/loader/entries

  • Check if GRUB_DEFAULT is set to 0 in /etc/default/grub file:

    # grep GRUB_DEFAULT /etc/default/grub 
   GRUB_DEFAULT=0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments