What is early kdump support and how do I configure it?

Resolution

What is early kdump support?

• early kdump is a new feature of the kdump mechanism to capture a vmcore of the booting kernel.
• In Red Hat Enterprise Linux 7 and below, the kdump service, which is responsible for setting up the system to produce a vmcore when a system crashes, starts much later during the boot sequence and typically when other services like sshd and the like are starting. This later timing prevents kdump from capturing vmcores when a system crashes before system services start.
• early kdump sets up the system much earlier in boot to create vmcores should the system crash before system services startup by loading the crash kernel and the crash kernel's initramfs into memory much earlier. For more details on the crash kernel and kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux

How is early kdump implemented?

• Two dracut modules are added in the kexec-tools package in order to load the crash kernel and its initramfs as early as possible;

   r8 # ls /usr/lib/dracut/modules.d/99earlykdump/
  early-kdump.sh  module-setup.sh
  
   r8 # dracut --list-modules | grep earlykdump
  earlykdump
  

• early kdump supports all the dump targets and configuration parameters supported by the normal kdump.

• By default early kdump is disabled.

Enabling early kdump

1. The system must first have kdump setup;

   • For details on configuring kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux.
   • For details on supported dump targets and network configurations for remote dump targets, please refer to the Specifying the Kdump Location of the above noted solution.

   • Ensure a kdump initramfs exists for the current kernel wherein the kdump service should be started if one does not exist;

      r8 # ls /boot/initramfs-`uname -r`kdump.img
     /boot/initramfs-4.18.0-305.17.1.el8_4.x86_64kdump.img
     
     # If nothing is listed from the above command, run the following command
      r8 # systemctl enable kdump && systemctl start kdump
     

2. Rebuild the initramfs of the booting kernel with early kdump support.

    r8 # dracut -f --add earlykdump 
   

3. Append the rd.earlykdump kernel boot parameter to kernelopts line in grub

   # Grab the current list of parameters
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200
   
   # Copy the above line and set the new option
    r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump"
   
   # Double check it is set
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump
   

   • For more information on setting kernel parameters, please refer to the knowledge-base solution, How to manually modify the boot parameter in grub before the system boots.

   • Enable early kdump for RHEL 9, please refer to Enabling early kdump for details.

4. Reboot the system for changes to take effect.

    r8 # reboot
   

   • Note The size of the initramfs of the normal kernel will be large because it includes the crash kernel (vmlinuz) and its initramfs;

     # lsinitrd /boot/initramfs-`uname -r`.img | grep ' boot'
     drwxr-xr-x   2 root     root            0 Oct 15 15:20 boot
     -rw-------   1 root     root     17966520 Oct 15 15:20 boot/initramfs-4.18.0-305.17.1.el8.x86_64kdump.img
     -rwxr-xr-x   1 root     root      7876752 Oct 15 15:20 boot/vmlinuz-4.18.0-305.17.1.el8.x86_64
     

5. To verify early kdump is enabled, review system logs (/var/log/messages or journalctl output depending on system configuration) for lines similar to the following;

   # journalctl -x | grep early-kdump 
   Dec 04 20:14:41 r8 dracut-cmdline[209]: early-kdump is enabled.
   Dec 04 20:14:42 r8 dracut-cmdline[209]: kexec: loaded early-kdump kernel
   

6. To optionally test early kdump functionality, refer to the knowledge-base solution, Test early kdump by passing custom kernel command line parameter early_panic

How to disable early kdump support?

1. Remove the kernel parameter 'rd.earlykdump' from grub;

   # Grab the current list of parameters
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump
   
   # Copy the above line and set the new option
    r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200"
   
   # Double check it is set
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200
   

2. Reboot the system for changes to take effect.

    r8 # reboot
   

3. To verify early kdump is disabled, review system logs;

    r8 # journalctl -x | grep early-kdump
   Dec 04 20:31:21 r8 dracut-cmdline[209]: early-kdump is disabled.
   

What is the limitation of early kdump support?

• Earlykdump is not enabled by default on the system for several reasons, as follows.

1. Early kdump depends on the normal kdump. Before enabling early kdump it is recommended to configure and start the normal kdump.
2. Enabling early kdump requires additional steps(as outlined in the beginning of this article), including a system reboot.
3. Early kdump involves copying the initramfs of the kdump kernel and vmlinuz inside the initramfs of the normal kernel, resulting in a larger size for the normal kernel's initramfs.
4. Early kdump inherits the settings of normal kdump. Any changes that require normal kdump to be rebuilt will also require the system initramfs to be rebuilt.
   • Note: Although the force_rebuild parameter in /etc/kdump.conf can automatically rebuild the kdump initramfs upon starting the kdump service, however this does not apply to early kdump, as the normal kernel initramfs cannot be rebuilt during the boot phase. In short the force_rebuild option is specific to only kdump initramfs file.
   • So kindly note that if system system configuration change during boot will affect the earlykdump and it may fail.
5. If the system updated to latest kernel and reboot the system with it, early kdump will be disabled by default. To enable it with the new kernel, you need to take the above steps again.
6. At present, early kdump doesn't support fadump.
7. Early kdump loads a crash kernel and initramfs at the beginning of the process in system's initramfs. A crash occurring earlier than that (e.g., in kernel initialization) cannot be captured even with early kdump.
8. For additional details please take a look at /usr/share/doc/kexec-tools/early-kdump-howto.txt file.