What is early kdump support and how do I configure it?

Resolution

What is early kdump support?

• early kdump is a new feature of the kdump mechanism to capture a vmcore of the booting kernel.
• In Red Hat Enterprise Linux 7 and below, the kdump service, which is responsible for setting up the system to produce a vmcore when a system crashes, starts much later during the boot sequence and typically when other services like sshd and the like are starting. This later timing prevents kdump from capturing vmcores when a system crashes before system services start.
• early kdump sets up the system much earlier in boot to create vmcores should the system crash before system services startup by loading the crash kernel and the crash kernel's initramfs into memory much earlier. For more details on the crash kernel and kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux

How is early kdump implemented?

• Two dracut modules are added in the kexec-tools package in order to load the crash kernel and its initramfs as early as possible;

   r8 # ls /usr/lib/dracut/modules.d/99earlykdump/
  early-kdump.sh  module-setup.sh
  
   r8 # dracut --list-modules | grep earlykdump
  earlykdump
  

• early kdump supports all the dump targets and configuration parameters supported by the normal kdump.

• By default early kdump is disabled.

Enabling early kdump

1. The system must first have kdump setup;

   • For details on configuring kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux.
   • For details on supported dump targets and network configurations for remote dump targets, please refer to the Specifying the Kdump Location of the above noted solution.

   • Ensure a kdump initramfs exists for the current kernel wherein the kdump service should be started if one does not exist;

      r8 # ls /boot/initramfs-`uname -r`kdump.img
     /boot/initramfs-4.18.0-305.17.1.el8_4.x86_64kdump.img
     
     # If nothing is listed from the above command, run the following command
      r8 # systemctl enable kdump && systemctl start kdump
     

2. Rebuild the initramfs of the booting kernel with early kdump support.

    r8 # dracut -f --add earlykdump 
   

3. Append the rd.earlykdump kernel boot parameter to kernelopts line in grub

   # Grab the current list of parameters
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200
   
   # Copy the above line and set the new option
    r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump"
   
   # Double check it is set
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump
   

   • For more information on setting kernel parameters, please refer to the knowledge-base solution, How to manually modify the boot parameter in grub before the system boots.

   • Enable early kdump for RHEL 9, please refer to Enabling early kdump for details.

4. Reboot the system for changes to take effect.

    r8 # reboot
   

   • Note The size of the initramfs of the normal kernel will be large because it includes the crash kernel (vmlinuz) and its initramfs;

     # lsinitrd /boot/initramfs-`uname -r`.img | grep ' boot'
     drwxr-xr-x   2 root     root            0 Oct 15 15:20 boot
     -rw-------   1 root     root     17966520 Oct 15 15:20 boot/initramfs-4.18.0-305.17.1.el8.x86_64kdump.img
     -rwxr-xr-x   1 root     root      7876752 Oct 15 15:20 boot/vmlinuz-4.18.0-305.17.1.el8.x86_64
     

5. To verify early kdump is enabled, review system logs (/var/log/messages or journalctl output depending on system configuration) for lines similar to the following;

   # journalctl -x | grep early-kdump 
   Dec 04 20:14:41 r8 dracut-cmdline[209]: early-kdump is enabled.
   Dec 04 20:14:42 r8 dracut-cmdline[209]: kexec: loaded early-kdump kernel
   

6. To optionally test early kdump functionality, refer to the knowledge-base solution, Test early kdump by passing custom kernel command line parameter early_panic

How to disable early kdump support?

1. Remove the kernel parameter 'rd.earlykdump' from grub;

   # Grab the current list of parameters
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump
   
   # Copy the above line and set the new option
    r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200"
   
   # Double check it is set
    r8 # grub2-editenv - list | grep kernelopts
   kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200
   

2. Reboot the system for changes to take effect.

    r8 # reboot
   

3. To verify early kdump is disabled, review system logs;

    r8 # journalctl -x | grep early-kdump
   Dec 04 20:31:21 r8 dracut-cmdline[209]: early-kdump is disabled.