What is early kdump support and how do I configure it?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • kexec-tools
  • dracut

Issue

  • What is early kdump support?
  • How to enable early kdump support?
  • How to verify if early kdump support is enabled?
  • How to disable early kdump support?
  • How to verify if early kdump support is disabled?
  • How to test early kdump support?
  • What is the limitation of early kdump support?
  • How to capture the kernel crash dump (vmcore) of the kernel at boot time?

Resolution

What is early kdump support?

  • Early kdump is a new feature of the kdump mechanism to capture the kernel crash dump of the booting kernel.

  • In earlier releases of RHEL(5/6/7), the kdump service starts too late during the boot sequence, so early crashes will have no chance to get kdump kernel booting, this will cause crash information to be lost.

  • Early Kdump stores the vmlinuz and initramfs of the crash kernel inside the initramfs of the booting kernel and load them directly into the reserved memory (crashkernel) during early boot stage.

  • Two dracut modules are added in the kexec-tools package in order to load crash kernel and initramfs as early as possible during the boot sequence to capture the kernel crash dump of the booting kernel.

/usr/lib/dracut/modules.d/99earlykdump/early-kdump.sh
/usr/lib/dracut/modules.d/99earlykdump/module-setup.sh

# dracut --list-modules | grep earlykdump
earlykdump

  • Early kdump supports all the dump targets and configuration parameters supported by the normal kdump.

  • By default early kdump is disabled.

How to enable early kdump support?

  • Configure the kdump service as per the steps provided in the kdump article.

  • Start the kdump service to make sure that the kdump initramfs is created.

Eg:

# systemctl enable kdump      
# systemctl start kdump
  • Rebuild the initramfs of the booting kernel with early kdump support.

Eg:

# dracut -f --add earlykdump
  • Append rd.earlykdump kernel boot parameter to 'kernelopts' line in /boot/grub2/grubenv file.

Eg:

# vi /boot/grub2/grubenv
kernelopts=root=/dev/mapper/rhel_rhel8-root ro crashkernel=auto resume=/dev/mapper/rhel_rhel8-swap rd.lvm.lv=rhel_rhel8/root rd.lvm.lv=rhel_rhel8/swap rd.earlykdump rhgb quiet
  • Reboot the system for changes to take effect.

Eg:

# reboot

Note: The size of the initramfs of the normal kernel will be large because it includes vmlinuz and initramfs of the crash kernel.

Eg:

# uname -r
4.18.0-32.el8.x86_64

# ls -l /boot/initramfs-`uname -r`.img
-rw-------. 1 root root 50562119 Dec  4 20:12 /boot/initramfs-4.18.0-32.el8.x86_64.img

# lsinitrd /boot/initramfs-`uname -r`.img | grep ' boot'
drwxr-xr-x   2 root     root            0 Oct 15 15:20 boot
-rw-------   1 root     root     17966520 Oct 15 15:20 boot/initramfs-4.18.0-32.el8.x86_64kdump.img
-rwxr-xr-x   1 root     root      7876752 Oct 15 15:20 boot/vmlinuz-4.18.0-32.el8.x86_64

How to verify if early kdump support is enabled?

  • Check the status of the early kdump support on the command prompt.

Eg:

# journalctl -x | grep early-kdump 
Dec 04 20:14:41 rhel8 dracut-cmdline[209]: early-kdump is enabled.
Dec 04 20:14:42 rhel8 dracut-cmdline[209]: kexec: loaded early-kdump kernel
  • If early kdump is successfully enabled then above messages will be logged.

How to disable early kdump support?

  • Remove 'rd.earlykdump' kernel boot parameter from 'kernelopts' option in /boot/grub2/grubenv file.

Eg:

# vi /boot/grub2/grubenv
kernelopts=root=/dev/mapper/rhel_rhel8-root ro crashkernel=auto resume=/dev/mapper/rhel_rhel8-swap rd.lvm.lv=rhel_rhel8/root rd.lvm.lv=rhel_rhel8/swap rhgb quiet
  • Reboot the system for changes to take effect.

Eg:

# reboot

How to verify if early kdump support is disabled?

  • Check the status of the early kdump support on the command prompt.

Eg:

# journalctl -x | grep early-kdump
Dec 04 20:31:21 rhel8 dracut-cmdline[209]: early-kdump is disabled.
  • If early kdump is successfully disabled then above message will be logged.

How to test early kdump support?

  • Create a custom unit file which starts before kdump.service and triggers panic using SysRq.

Eg:

# touch /etc/systemd/system/sysrq.service

# chmod 664 /etc/systemd/system/sysrq.service

# cat /etc/systemd/system/sysrq.service
[Unit]
Description=SysRq Service
Before=kdump.service

[Service]
ExecStart=/usr/local/sysrq.sh
Type=simple

[Install]
WantedBy=default.target

# cat /usr/local/sysrq.sh
#!/bin/bash
/usr/bin/echo c > /proc/sysrq-trigger

# chmod +x /usr/local/sysrq.sh

# systemctl daemon-reload

# systemctl enable sysrq.service

Caution: Don't start the sysrq.service otherwise, the system will crash immediately.

  • Reboot the system to capture the kernel crash dump (vmcore) by early kdump.

Eg:

# reboot
  • Boot the system in rescue mode using 'systemd.unit=rescue.target' and disable the service 'sysrq.service' at boot time.
# systemctl disable sysrq.service
  • Verify the kernel crash dump inside the dump target location.

Eg:

# tree /var/crash/
/var/crash/
└── 127.0.0.1-2018-12-04-20:23:40
    ├── vmcore
    └── vmcore-dmesg.txt

1 directory, 2 files

# cat vmcore-dmesg.txt
...
[    5.933424] sysrq: SysRq : Trigger a crash
[    5.933479] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[    5.933510] PGD 0 P4D 0 
[    5.933525] Oops: 0002 [#1] SMP PTI
[    5.933543] CPU: 1 PID: 813 Comm: echo Kdump: loaded Not tainted 4.18.0-32.el8.x86_64 #1
[    5.933571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014
[    5.933624] RIP: 0010:sysrq_handle_crash+0x12/0x20

What is the limitation of early kdump support?

  • At present, early kdump doesn't support fadump.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

3 Comments

Log in to comment
Xinhua Li's picture Red Hat Active Contributor 141 points
21 September 2019 3:16 PM

For this script sysrq.sh, we can use similar as below. In this case, we can check if we have suicide keyword in Kernel command line. (We can add it temporary via grub command line and trigger vmcore one time. So we do not need to use 'systemd.unit=rescue.target' to disable the sysrq.service)

#!/bin/bash
if [ `cat /proc/cmdline |grep suicide >/dev/null && echo $?` ]
    then echo "## Trigger kdump" >>/var/log/messages && echo "c" > /proc/sysrq-trigger
fi
AM Newbie 17 points
8 November 2019 10:13 AM

Many issues with this script :

- Using backticks is deprecated since many years. You must use the idiom "$( )" instead of "` `"
- grep'ing for a word is a very bad practice and is dangerous when associated with such a feature. You must specify very precisely the word with boundaries
- uuoc
- If you use bash, use smart test [[  instead of old [
- Don't use echo $? within a test [[. Use the command directly
- Don't use "&&" when you just want to chain commands, whatever their result

#!/bin/bash
if grep -w suicide /proc/cmdline >/dev/null; then
  echo "## Trigger kdump" | tee -a /var/log/messages; echo c > /proc/sysrq-trigger
fi

if your version of grep don't implements "-w" option, you can implement it with something like that, to match with either a space or end-of-line:

grep -e ' suicide$' -e ' suicide ' /proc/cmdline
AM Newbie 17 points
8 November 2019 8:58 AM

== WARNING == Do not edit /boot/grub2/grubenv manually. The file must be exactly 1024 bytes, and this size is checked by grub2-editenv If you edit it manually, you'll get errors like in BZ #1683128 (https://bugzilla.redhat.com/show_bug.cgi?id=1683128) :

grub2-editenv: error: environment block too small.

You should not edit it manually. Instead:

- In file "/etc/default/grub", add "rd.earlykdump" to the variable "GRUB_CMDLINE_LINUX"
- Regenerate the configuration with the following command:
  # If BIOS:  grub2-mkconfig -o /boot/grub2/grub.cfg
  # If UEFI: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg