What is early kdump support and how do I configure it?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8 and above
  • kexec-tools
  • Note early kdump currently does not support POWERPC's fadump

Issue

  • What is early kdump support?
  • How to enable early kdump support
  • How to verify if early kdump support is enabled
  • How to disable early kdump support
  • How to verify if early kdump support is disabled
  • How to test early kdump support
  • What is the limitation of early kdump support?
  • How to capture the kernel crash dump (vmcore) of the kernel at boot time

Resolution

What is early kdump support?

  • early kdump is a new feature of the kdump mechanism to capture a vmcore of the booting kernel.
  • In Red Hat Enterprise Linux 7 and below, the kdump service, which is responsible for setting up the system to produce a vmcore when a system crashes, starts much later during the boot sequence and typically when other services like sshd and the like are starting. This later timing prevents kdump from capturing vmcores when a system crashes before system services start.
  • early kdump sets up the system much earlier in boot to create vmcores should the system crash before system services startup by loading the crash kernel and the crash kernel's initramfs into memory much earlier. For more details on the crash kernel and kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux

How is early kdump implemented?

  • Two dracut modules are added in the kexec-tools package in order to load the crash kernel and its initramfs as early as possible;

     r8 # ls /usr/lib/dracut/modules.d/99earlykdump/
early-kdump.sh  module-setup.sh

 r8 # dracut --list-modules | grep earlykdump
earlykdump

  • early kdump supports all the dump targets and configuration parameters supported by the normal kdump.

  • By default early kdump is disabled.

Enabling early kdump

  1. The system must first have kdump setup;

    • For details on configuring kdump, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux.
    • For details on supported dump targets and network configurations for remote dump targets, please refer to the Specifying the Kdump Location of the above noted solution.

    • Ensure a kdump initramfs exists for the current kernel wherein the kdump service should be started if one does not exist;

       r8 # ls /boot/initramfs-`uname -r`kdump.img
/boot/initramfs-4.18.0-305.17.1.el8_4.x86_64kdump.img

# If nothing is listed from the above command, run the following command
 r8 # systemctl enable kdump && systemctl start kdump

  2. Rebuild the initramfs of the booting kernel with early kdump support.

     r8 # dracut -f --add earlykdump

  3. Append the rd.earlykdump kernel boot parameter to kernelopts line in grub

    # Grab the current list of parameters
 r8 # grub2-editenv - list | grep kernelopts
kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200

# Copy the above line and set the new option
 r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump"

# Double check it is set
 r8 # grub2-editenv - list | grep kernelopts
kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump

  4. Reboot the system for changes to take effect.

     r8 # reboot

    • Note The size of the initramfs of the normal kernel will be large because it includes the crash kernel (vmlinuz) and its initramfs;

      # lsinitrd /boot/initramfs-`uname -r`.img | grep ' boot'
drwxr-xr-x   2 root     root            0 Oct 15 15:20 boot
-rw-------   1 root     root     17966520 Oct 15 15:20 boot/initramfs-4.18.0-305.17.1.el8.x86_64kdump.img
-rwxr-xr-x   1 root     root      7876752 Oct 15 15:20 boot/vmlinuz-4.18.0-305.17.1.el8.x86_64

  5. To verify early kdump is enabled, review system logs (/var/log/messages or journalctl output depending on system configuration) for lines similar to the following;

    # journalctl -x | grep early-kdump 
Dec 04 20:14:41 r8 dracut-cmdline[209]: early-kdump is enabled.
Dec 04 20:14:42 r8 dracut-cmdline[209]: kexec: loaded early-kdump kernel

  6. To optionally test early kdump functionality, refer to the knowledge-base solution, Test early kdump by passing custom kernel command line parameter early_panic

How to disable early kdump support?

  1. Remove the kernel parameter 'rd.earlykdump' from grub;

    # Grab the current list of parameters
 r8 # grub2-editenv - list | grep kernelopts
kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump

# Copy the above line and set the new option
 r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200"

# Double check it is set
 r8 # grub2-editenv - list | grep kernelopts
kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200

  2. Reboot the system for changes to take effect.

     r8 # reboot

  3. To verify early kdump is disabled, review system logs;

     r8 # journalctl -x | grep early-kdump
Dec 04 20:31:21 r8 dracut-cmdline[209]: early-kdump is disabled.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

3 Comments

Log in to comment
Xinhua Li's picture Red Hat Active Contributor 206 points
21 September 2019 3:16 PM

For this script sysrq.sh, we can use similar as below. In this case, we can check if we have suicide keyword in Kernel command line. (We can add it temporary via grub command line and trigger vmcore one time. So we do not need to use 'systemd.unit=rescue.target' to disable the sysrq.service)

#!/bin/bash
if [ `cat /proc/cmdline |grep suicide >/dev/null && echo $?` ]
    then echo "## Trigger kdump" >>/var/log/messages && echo "c" > /proc/sysrq-trigger
fi
AM Newbie 17 points
8 November 2019 10:13 AM

Many issues with this script :

- Using backticks is deprecated since many years. You must use the idiom "$( )" instead of "` `"
- grep'ing for a word is a very bad practice and is dangerous when associated with such a feature. You must specify very precisely the word with boundaries
- uuoc
- If you use bash, use smart test [[  instead of old [
- Don't use echo $? within a test [[. Use the command directly
- Don't use "&&" when you just want to chain commands, whatever their result

#!/bin/bash
if grep -w suicide /proc/cmdline >/dev/null; then
  echo "## Trigger kdump" | tee -a /var/log/messages; echo c > /proc/sysrq-trigger
fi

if your version of grep don't implements "-w" option, you can implement it with something like that, to match with either a space or end-of-line:

grep -e ' suicide$' -e ' suicide ' /proc/cmdline
AM Newbie 17 points
8 November 2019 8:58 AM

== WARNING == Do not edit /boot/grub2/grubenv manually. The file must be exactly 1024 bytes, and this size is checked by grub2-editenv If you edit it manually, you'll get errors like in BZ #1683128 (https://bugzilla.redhat.com/show_bug.cgi?id=1683128) :

grub2-editenv: error: environment block too small.

You should not edit it manually. Instead:

- In file "/etc/default/grub", add "rd.earlykdump" to the variable "GRUB_CMDLINE_LINUX"
- Regenerate the configuration with the following command:
  # If BIOS:  grub2-mkconfig -o /boot/grub2/grub.cfg
  # If UEFI: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg