What is early kdump support and how do I configure it?
Environment
- Red Hat Enterprise Linux 8 and above
kexec-tools
- Note
early kdump
currently does not support POWERPC'sfadump
Issue
- What is
early kdump
support? - How to enable
early kdump
support - How to verify if
early kdump
support is enabled - How to disable
early kdump
support - How to verify if
early kdump
support is disabled - How to test
early kdump
support - What is the limitation of
early kdump
support? - How to capture the kernel crash dump (vmcore) of the kernel at boot time
Resolution
What is early kdump
support?
early kdump
is a new feature of thekdump
mechanism to capture a vmcore of the booting kernel.- In Red Hat Enterprise Linux 7 and below, the
kdump
service, which is responsible for setting up the system to produce a vmcore when a system crashes, starts much later during the boot sequence and typically when other services likesshd
and the like are starting. This later timing preventskdump
from capturing vmcores when a system crashes before system services start. early kdump
sets up the system much earlier in boot to create vmcores should the system crash before system services startup by loading the crash kernel and the crash kernel'sinitramfs
into memory much earlier. For more details on the crash kernel andkdump
, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux
How is early kdump
implemented?
-
Two
dracut
modules are added in thekexec-tools
package in order to load the crash kernel and itsinitramfs
as early as possible;r8 # ls /usr/lib/dracut/modules.d/99earlykdump/ early-kdump.sh module-setup.sh r8 # dracut --list-modules | grep earlykdump earlykdump
-
early kdump
supports all the dump targets and configuration parameters supported by the normal kdump. -
By default
early kdump
is disabled.
Enabling early kdump
-
The system must first have
kdump
setup;- For details on configuring
kdump
, please refer to the knowledge-base solution, How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux. - For details on supported dump targets and network configurations for remote dump targets, please refer to the Specifying the Kdump Location of the above noted solution.
-
Ensure a
kdump
initramfs
exists for the current kernel wherein thekdump
service should be started if one does not exist;r8 # ls /boot/initramfs-`uname -r`kdump.img /boot/initramfs-4.18.0-305.17.1.el8_4.x86_64kdump.img # If nothing is listed from the above command, run the following command r8 # systemctl enable kdump && systemctl start kdump
- For details on configuring
-
Rebuild the
initramfs
of the booting kernel withearly kdump
support.r8 # dracut -f --add earlykdump
-
Append the
rd.earlykdump
kernel boot parameter tokernelopts
line ingrub
# Grab the current list of parameters r8 # grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 # Copy the above line and set the new option r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump" # Double check it is set r8 # grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump
-
For more information on setting kernel parameters, please refer to the knowledge-base solution, How to manually modify the boot parameter in grub before the system boots.
-
Enable early kdump for RHEL 9, please refer to Enabling early kdump for details.
-
-
Reboot the system for changes to take effect.
r8 # reboot
-
Note The size of the
initramfs
of the normal kernel will be large because it includes the crash kernel (vmlinuz
) and itsinitramfs
;# lsinitrd /boot/initramfs-`uname -r`.img | grep ' boot' drwxr-xr-x 2 root root 0 Oct 15 15:20 boot -rw------- 1 root root 17966520 Oct 15 15:20 boot/initramfs-4.18.0-305.17.1.el8.x86_64kdump.img -rwxr-xr-x 1 root root 7876752 Oct 15 15:20 boot/vmlinuz-4.18.0-305.17.1.el8.x86_64
-
-
To verify
early kdump
is enabled, review system logs (/var/log/messages
orjournalctl
output depending on system configuration) for lines similar to the following;# journalctl -x | grep early-kdump Dec 04 20:14:41 r8 dracut-cmdline[209]: early-kdump is enabled. Dec 04 20:14:42 r8 dracut-cmdline[209]: kexec: loaded early-kdump kernel
-
To optionally test
early kdump
functionality, refer to the knowledge-base solution, Test early kdump by passing custom kernel command line parameter early_panic
How to disable early kdump
support?
-
Remove the kernel parameter 'rd.earlykdump' from
grub
;# Grab the current list of parameters r8 # grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200 rd.earlykdump # Copy the above line and set the new option r8 # grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200" # Double check it is set r8 # grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=tty0 console=ttyS0,115200
-
Reboot the system for changes to take effect.
r8 # reboot
-
To verify
early kdump
is disabled, review system logs;r8 # journalctl -x | grep early-kdump Dec 04 20:31:21 r8 dracut-cmdline[209]: early-kdump is disabled.
What is the limitation of early kdump
support?
- Earlykdump is not enabled by default on the system for several reasons, as follows.
- Early kdump depends on the normal kdump. Before enabling early kdump it is recommended to configure and start the normal kdump.
- Enabling early kdump requires additional steps(as outlined in the beginning of this article), including a system reboot.
- Early kdump involves copying the initramfs of the kdump kernel and vmlinuz inside the initramfs of the normal kernel, resulting in a larger size for the normal kernel's initramfs.
- Early kdump inherits the settings of normal kdump. Any changes that require normal kdump to be rebuilt will also require the system initramfs to be rebuilt.
- Note: Although the
force_rebuild
parameter in/etc/kdump.conf
can automatically rebuild the kdump initramfs upon starting the kdump service, however this does not apply to early kdump, as the normal kernel initramfs cannot be rebuilt during the boot phase. In short theforce_rebuild
option is specific to only kdump initramfs file. - So kindly note that if system system configuration change during boot will affect the earlykdump and it may fail.
- Note: Although the
- If the system updated to latest kernel and reboot the system with it, early kdump will be disabled by default. To enable it with the new kernel, you need to take the above steps again.
- At present, early kdump doesn't support fadump.
- Early kdump loads a crash kernel and initramfs at the beginning of the process in system's initramfs. A crash occurring earlier than that (e.g., in kernel initialization) cannot be captured even with early kdump.
- For additional details please take a look at
/usr/share/doc/kexec-tools/early-kdump-howto.txt
file.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments