Menu Close

Chapter 11. Configuring kdump on the command line

The memory for kdump is reserved during the system boot. The memory size is configured in the system’s Grand Unified Bootloader (GRUB) 2 configuration file. The memory size depends on the crashkernel= value specified in the configuration file and the size of the system’s physical memory.

11.1. Configuring kdump memory usage

The kexec-tools package maintains the default crashkernel memory reservation values. The kdump service uses the default value to reserve the crashkernel memory for each kernel. This default value can also form the reference base value to estimate the required memory size when setting crashkernel value manually. The minimum size of the crash kernel may vary depending on the hardware and machine specifications.

The automatic memory allocation for kdump also varies based on the system hardware architecture and available memory size. For example, on AMD and Intel 64-bit architectures, the crashkernel default parameters work only when the available memory is more than 1GB. By default, kexec-tools configures the following memory reserves on AMD64 and Intel 64-bit architectures:

crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M

You can also run kdumpctl estimate to query a rough estimate value without triggering a crash. The estimated crashkernel value may not be accurate and can serve as a reference to set an appropriate crashkernel value.

Note

The crashkernel=auto option in the boot command line is no longer supported on RHEL 9 and later releases.

Prerequisites

  • Root privileges
  • Fulfilled kdump requirements for configurations and targets.

Procedure

  1. Configure default value for crashkernel: 

    # kdumpctl reset-crashkernel --kernel=ALL

  2. To use a custom crashkernel value:

    • Configure the required memory reserve. 

      # crashkernel=192M

      The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4GB. If the total amount of memory is more than 4GB, 256MB is reserved for kdump instead.

      • (Optional) Offset the reserved memory:

        Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax: 

        # crashkernel=192M@16M

        The example above reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example, crashkernel=1G-4G:192M,2G-64G:256M@16M.

    • Update the bootloader configuration: 

      # grubby --update-kernel ALL --args "crashkernel=<CUSTOM-VALUE>”

  3. Reboot for changes to take effect: 

    # reboot

Verification

  1. Activate the sysrq key to boot into the kdump kernel: 

    # echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger

    This forces the Linux kernel to crash and copy the address-YYYY-MM-DD-HH:MM:SS/vmcore file to the target location specified in the configuration file.

  2. Verify that the vmcore file is dumped in the target as specified in the /etc/kdump.conf file. 

    $ ls /var/crash/127.0.0.1-2022-01-18-0

/var/crash/127.0.0.1-2022-01-18-05:23:10':
kexec-dmesg.log  vmcore  vmcore-dmesg.txt

    In this example, the kernel saves the vmcore in the default target directory, /var/crash/.

11.2. Configuring the kdump target

When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. Only one of these options can be set at a time, and the default behavior is to store the vmcore file in the /var/crash/ directory of the local file system.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.

Procedure

  • To store the vmcore file in /var/crash/ directory of the local file system, edit the /etc/kdump.conf file and specify the path: 

    # path /var/crash

    The option path /var/crash represents the path to the file system in which kdump saves the vmcore file. When you specify a dump target in the /etc/kdump.conf file, then the path is relative to the specified dump target.

    If you do not specify a dump target in the /etc/kdump.conf file, then the path represents the absolute path from the root directory. Depending on what is mounted in the current system, the dump target and the adjusted dump path are taken automatically.

Warning

kdump saves the vmcore file in /var/crash/var/crash directory, when the dump target is mounted at /var/crash and the option path is also set as /var/crash in the /etc/kdump.conf file. For example, in the following instance, the ext4 file system is already mounted at /var/crash and the path are set as /var/crash: 

grep -v ^# etc/kdump.conf | grep -v ^$
ext4 /dev/mapper/vg00-varcrashvol
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31

This results in the /var/crash/var/crash path. To solve this problem, use the option path / instead of path /var/crash

  • To change the local directory in which the core dump is to be saved, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #path /var/crash line.

    2. Replace the value with the intended directory path. For example: 

      # path /usr/local/cores
      Important

      In RHEL 9, the directory defined as the kdump target using the path directive must exist when the kdump systemd service is started - otherwise the service fails. This behavior is different from earlier releases of RHEL, where the directory was being created automatically if it did not exist when starting the service.

  • To write the file to a different partition, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #ext4 line, depending on your choice.

      • device name (the #ext4 /dev/vg/lv_kdump line)
      • file system label (the #ext4 LABEL=/boot line)
      • UUID (the #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 line)

    2. Change the file system type as well as the device name, label or UUID to the desired values. For example: 

      # ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
      Important

      It is recommended to specify storage devices using a LABEL= or UUID=. Disk device names such as /dev/sda3 are not guaranteed to be consistent across reboot.

      Important

      When dumping to Direct Access Storage Device (DASD) on IBM Z hardware, it is essential that the dump devices are correctly specified in /etc/dasd.conf before proceeding.

  • To write the dump directly to a device:

    1. Remove the hash sign ("#") from the beginning of the #raw /dev/vg/lv_kdump line.

    2. Replace the value with the intended device name. For example: 

      # raw /dev/sdb1

  • To store the dump to a remote machine using the NFS protocol:

    1. Remove the hash sign ("#") from the beginning of the #nfs my.server.com:/export/tmp line.

    2. Replace the value with a valid hostname and directory path. For example: 

      # nfs penguin.example.com:/export/cores

  • To store the dump to a remote machine using the SSH protocol:

    1. Remove the hash sign ("#") from the beginning of the #ssh user@my.server.com line.
    2. Replace the value with a valid username and hostname.

    3. Include your SSH key in the configuration.

      • Remove the hash sign from the beginning of the #sshkey /root/.ssh/kdump_id_rsa line.

      • Change the value to the location of a key valid on the server you are trying to dump to. For example: 

        # ssh john@penguin.example.com
sshkey /root/.ssh/mykey

11.3. Configuring the core collector

The kdump service uses the core_collector program to capture the vmcore image. In RHEL, the makedumpfile utility is the default core collector.

makedumpfile is a dump program that helps to copy only the required pages using various dump levels and compress the size of a dump file.

makedumpfile is a dump program that helps to copy only necessary pages using various dump levels and compress the size of a dump file.

Using makedumpfile, you can create a small size dump file either by compressing dump data or by excluding pages or both. It needs the first kernel debug information to distinguish the not necessary pages by analyzing how the first kernel uses the memory.

Syntax

core_collector makedumpfile -z -d 31 --message-level 1

Options

  • -c, -l, -z, or -p: specifies the compress dump file format by each page when you use one of these options: -c for zlib, -l for lzo, -z for zstd, or -p for snappy.
  • -d (dump_level): excludes pages so that they do not get copied to the dump file.

  • --message-level : specifies the message types.

    Using --message-level, you can restrict the outputs to print. For example, specifying 7 as the message level prints common messages and error messages. The maximum value for --message_level is 31.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.

Procedure

  1. As root user, edit the /etc/kdump.conf configuration file to remove the hash sign ("#") from the beginning of the following command: 

    #core_collector makedumpfile -z -d 31 --message-level 1

  2. To enable dump file compression, specify one of the makedumpfile options: 

    core_collector makedumpfile -z -d 31 --message-level 1

    where,

    • -z specifies the dump compressed file format.
    • -d specifies dump level as 31.
    • --message-level specifies message level as 1.

Also, consider the following example that uses -l:

  • To compress a dump file using -l: 

    core_collector makedumpfile -l -d 31 --message-level 1

Additional resources

  • makedumpfile(8) manual page

11.4. Configuring the kdump default failure responses

By default, when kdump fails to create a crash dump file at the configured target location, the system reboots and the dump is lost in the process. To change this behavior, follow the procedure below.

Prerequisites

  • Root permissions.
  • Fulfilled requirements for kdump configurations and targets.

Procedure

  1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

  2. Replace the value with a desired action. 

    # failure_action poweroff

11.5. Configuration file for kdump

The configuration file for kdump kernel is /etc/sysconfig/kdump. This file controls the kdump kernel command line parameters.

For most configurations, use the default options. However, in some scenarios you might need to modify certain parameters to control the kdump kernel behavior. For example, modifying to append the kdump kernel command-line to obtain a detailed debugging output.

This section covers information on modifying the KDUMP_COMMANDLINE_REMOVE and KDUMP_COMMANDLINE_APPEND options for kdump. For information on additional configuration options refer to Documentation/admin-guide/kernel-parameters.txt or the /etc/sysconfig/kdump file.

  • KDUMP_COMMANDLINE_REMOVE

    This option removes arguments from the current kdump command line. It removes parameters that may cause kdump errors or kdump kernel boot failures. These parameters may have been parsed from the previous KDUMP_COMMANDLINE process or inherited from the /proc/cmdline file. When this variable is not configured, it inherits all values from the /proc/cmdline file. Configuring this option also provides information that is helpful in debugging an issue.

    Example

    To remove certain arguments, add them to KDUMP_COMMANDLINE_REMOVE as follows: 

    # KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"

  • KDUMP_COMMANDLINE_APPEND

    This option appends arguments to the current command line. These arguments may have been parsed by the previous KDUMP_COMMANDLINE_REMOVE variable.

    For the kdump kernel, disabling certain modules such as mce, cgroup, numa, hest_disable can help prevent kernel errors. These modules may consume a significant portion of the kernel memory reserved for kdump or cause kdump kernel boot failures.

    Example

    To disable memory cgroups on the kdump kernel command line, run the command as follows: 

    # KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory"

Additional resources

  • Documentation/admin-guide/kernel-parameters.txt file
  • /etc/sysconfig/kdump file

11.6. Enabling and disabling the kdump service

To start the kdump service at boot time, follow the procedure below.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.
  • All configurations for installing kdump are set up according to your needs.

Procedure

  1. To enable the kdump service, use the following command: 

    # systemctl enable kdump.service

    This enables the service for multi-user.target.

  2. To start the service in the current session, use the following command: 

    # systemctl start kdump.service

  3. To stop the kdump service, type the following command: 

    # systemctl stop kdump.service

  4. To disable the kdump service, execute the following command: 

    # systemctl disable kdump.service
Warning

It is recommended to set kptr_restrict=1 as default. When kptr_restrict is set to (1) as default, the kdumpctl service loads the crash kernel even if Kernel Address Space Layout (KASLR) is enabled or not enabled.

Troubleshooting step

When kptr_restrict is not set to (1), and if KASLR is enabled, the contents of /proc/kore file are generated as all zeros. Consequently, the kdumpctl service fails to access the /proc/kcore and load the crash kernel.

To work around this problem, the kexec-kdump-howto.txt file displays a warning message, which specifies to keep the recommended setting as kptr_restrict=1.

To ensure that kdumpctl service loads the crash kernel, verify that:

  • Kernel kptr_restrict=1 in the sysctl.conf file.

11.7. Testing the kdump configuration

You can test that the crash dump process works and is valid before the machine enters production.

Warning

The commands below cause the kernel to crash. Use caution when following these steps, and never carelessly use them on active production system.

Procedure

  1. Reboot the system with kdump enabled.

  2. Make sure that kdump is running: 

    # systemctl is-active kdump
active

  3. Force the Linux kernel to crash: 

    # echo 1 &gt; /proc/sys/kernel/sysrq
# echo c &gt; /proc/sysrq-trigger
    Warning

    The command above crashes the kernel, and a reboot is required.

    Once booted again, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file (by default to /var/crash/).

    Note

    This action confirms the validity of the configuration. Also it is possible to use this action to record how long it takes for a crash dump to complete with a representative work-load.

11.8. Preventing kernel drivers from loading for kdump

This section explains how to prevent the capture kernel from loading certain kernel drivers using the /etc/sysconfig/kdump configuration file. You can prevent the kdump initramfs from loading the specified kernel module. To achieve this, you need to put the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file. This helps to prevent the out-of-memory (oom) killer or other crash kernel failures.

You can append the KDUMP_COMMANDLINE_APPEND= variable using one of the following configuration options:

  • rd.driver.blacklist=<modules>
  • modprobe.blacklist=<modules>

Procedure

  1. Select a kernel module that you intend to block from loading. 

    $ lsmod

Module                  Size  Used by
fuse                  126976  3
xt_CHECKSUM            16384  1
ipt_MASQUERADE         16384  1
uinput                 20480  1
xt_conntrack           16384  1

    The lsmod command displays a list of modules that are loaded to the currently running kernel.

  2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file. 

    # KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

    Also,consider the following example using the modprobe.blacklist=<modules> configuration option. 

    # KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

  3. Restart the kdump service. 

    # systemctl restart kdump

Additional resources

  • dracut.cmdline manual page

11.9. Running kdump on systems with encrypted disk

When you run a Linux Unified Key Setup (LUKS) encrypted partition, the system require certain amount of available memory. If the system has less than the required amount of available memory, the systemd-cryptsetup service fails to mount the partition. As a result, capturing the vmcore file to an encrypted target location fails in the second kernel (capture kernel).

The kdumpctl estimate command helps you estimate the amount of memory you need for kdump. It prints the recommended crashkernel value, which is the most suitable memory size required for kdump.

The recommended crashkernel value is calculated based on the current kernel size, kernel modules, initramfs, and the LUKS encrypted target memory requirement.

In case you use the custom crashkernel option, kdumpctl estimate prints the LUKS required size value. The value is the memory size required for LUKS encrypted target.

Procedure

  1. Print the estimate crashkernel value: 

    # kdumpctl estimate

Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
   Reserved crashkernel:    256M
   Recommended crashkernel: 652M
   Kernel image size:   47M
   Kernel modules size: 8M
   Initramfs size:      20M
   Runtime reservation: 64M
   LUKS required size:  512M
   Large modules: none
   WARNING: Current crashkernel size is lower than recommended size 652M.

  2. Configure the amount of required memory by increasing crashkernel to the desired value. 

    # grubby –args=”crashkernel=652M” --update-kernel=ALL

  3. Reboot for changes to take effect. 

    # reboot
Note

If the kdump service still fails to save the dump file to the encrypted target, increase the crashkernel value gradually to configure an appropriate amount of memory.