Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 11. Configuring kdump on the command line

The memory for kdump is reserved during the system boot. The memory size is configured in the system’s Grand Unified Bootloader (GRUB) 2 configuration file. The memory size depends on the crashkernel= value specified in the configuration file and the size of the system’s physical memory.

11.1. Configuring kdump memory usage

The crashkernel= option can be defined in multiple ways. You can specify the crashkernel= value or configure the auto option. The crashkernel=auto boot option, reserves memory automatically, depending on total amount of the system’s physical memory. When configured, the kernel will automatically reserve an appropriate amount of required memory for the kdump kernel. This helps to prevent Out-of-Memory (OOM) error occurrences.

Note

The automatic memory allocation for kdump varies based on system hardware architecture and available memory size.

For example, on AMD64 and Intel 64, the crashkernel=auto parameter works only when the available memory is more than 1GB and 64-bit ARM architecture and IBM Power Systems have available memory more than 2GB.

If the system has less than the minimum memory threshold for automatic allocation, you can configure the amount of reserved memory manually.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets

Procedure

  1. Edit the /etc/default/grub file using the root permissions.
  2. Set the crashkernel= option to the required value.

    For example, to reserve 128 MB of memory, use the following:

    crashkernel=128M

    Alternatively, you can set the amount of reserved memory to a variable depending on the total amount of installed memory. The syntax for memory reservation into a variable is crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

    crashkernel=512M-2G:64M,2G-:128M

    The above example reserves 64 MB of memory if the total amount of system memory is 512 MB or higher and lower than 2 GB. If the total amount of memory is more than 2 GB, 128 MB is reserved for kdump instead.

    • Offset the reserved memory.

      Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

      crashkernel=128M@16M

      The example above means that kdump reserves 128 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump offsets the reserved memory automatically. This syntax can also be used when setting a variable memory reservation as described above; in this case, the offset is always specified last (for example, crashkernel=512M-2G:64M,2G-:128M@16M).

  3. Use the following command to update the GRUB2 configuration file:

    # grub2-mkconfig -o /boot/grub2/grub.cfg
Note

The alternative way to configure memory for kdump is to append the crashkernel=<SOME_VALUE> parameter to the kernelopts variable with the grub2-editenv which will update all of your boot entries. Or you can use the grubby utility to update kernel command line parameters of just one entry.

11.2. Configuring the kdump target

When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. Only one of these options can be set at a time, and the default behavior is to store the vmcore file in the /var/crash/ directory of the local file system.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.

Procedure

  • To store the vmcore file in /var/crash/ directory of the local file system, edit the /etc/kdump.conf file and specify the path:

    path /var/crash

    The option path /var/crash represents the path to the file system in which kdump saves the vmcore file. When you specify a dump target in the /etc/kdump.conf file, then the path is relative to the specified dump target.

    If you do not specify a dump target in the /etc/kdump.conf file, then the path represents the absolute path from the root directory. Depending on what is mounted in the current system, the dump target and the adjusted dump path are taken automatically.

Warning

kdump saves the vmcore file in /var/crash/var/crash directory, when the dump target is mounted at /var/crash and the option path is also set as /var/crash in the /etc/kdump.conf file. For example, in the following instance, the ext4 file system is already mounted at /var/crash and the path are set as /var/crash:

grep -v ^# etc/kdump.conf | grep -v ^$
ext4 /dev/mapper/vg00-varcrashvol
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31

This results in the /var/crash/var/crash path. To solve this problem, use the option path / instead of path /var/crash

  • To change the local directory in which the core dump is to be saved, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #path /var/crash line.
    2. Replace the value with the intended directory path. For example:

      path /usr/local/cores
      Important

      In Red Hat Enterprise Linux 8, the directory defined as the kdump target using the path directive must exist when the kdump systemd service is started - otherwise the service fails. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service.

  • To write the file to a different partition, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #ext4 line, depending on your choice.

      • device name (the #ext4 /dev/vg/lv_kdump line)
      • file system label (the #ext4 LABEL=/boot line)
      • UUID (the #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 line)
    2. Change the file system type as well as the device name, label or UUID to the desired values. For example:

      ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
      Important

      It is recommended to specify storage devices using a LABEL= or UUID=. Disk device names such as /dev/sda3 are not guaranteed to be consistent across reboot.

      Important

      When dumping to Direct Access Storage Device (DASD) on IBM Z hardware, it is essential that the dump devices are correctly specified in /etc/dasd.conf before proceeding.

  • To write the dump directly to a device:

    1. Remove the hash sign ("#") from the beginning of the #raw /dev/vg/lv_kdump line.
    2. Replace the value with the intended device name. For example:

      raw /dev/sdb1
  • To store the dump to a remote machine using the NFS protocol:

    1. Remove the hash sign ("#") from the beginning of the #nfs my.server.com:/export/tmp line.
    2. Replace the value with a valid hostname and directory path. For example:

      nfs penguin.example.com:/export/cores
  • To store the dump to a remote machine using the SSH protocol:

    1. Remove the hash sign ("#") from the beginning of the #ssh user@my.server.com line.
    2. Replace the value with a valid username and hostname.
    3. Include your SSH key in the configuration.

      • Remove the hash sign from the beginning of the #sshkey /root/.ssh/kdump_id_rsa line.
      • Change the value to the location of a key valid on the server you are trying to dump to. For example:

        ssh john@penguin.example.com
        sshkey /root/.ssh/mykey

Additional resources

11.3. Configuring the core collector

The kdump service uses a core_collector program to capture the vmcore image. In RHEL, the makedumpfile utility is the default core collector .

makedumpfile is a dump program that helps to compress the size of a dump file and copy only necessary pages using various dump levels.

makedumpfile makes a small dump file either by compressing dump data or by excluding unnecessary pages or both. It needs the first kernel debug information to understand how first kernel uses the memory. This helps to detect the necessary pages.

Syntax

core_collector makedumpfile -l --message-level 1 -d 31

Options

  • -c, -l or -p: specify compress dump file format by each page using either, zlib for -c option, lzo for -l option or snappy for -p option.
  • -d (dump_level): excludes pages so that they are not copied to the dump file.
  • --message-level : specify the message types. You can restrict outputs printed by specifying message_level with this option. For example, specifying 7 as message_level prints common messages and error messages. The maximum value of message_level is 31

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.

Procedure

  1. As root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.
  2. To enable dump file compression, execute:
core_collector makedumpfile -l --message-level 1 -d 31

where,

  • -l specifies the dump compressed file format.
  • -d specifies dump level as 31.
  • --message-level specifies message level as 1.

Also, consider following examples using options -c and -p:

  • To compress dump file using -c:
core_collector makedumpfile -c -d 31 --message-level 1
  • To compress dump file using -p:
core_collector makedumpfile -p -d 31 --message-level 1

Additional resources

  • makedumpfile(8) manual page

11.4. Configuring the kdump default failure responses

By default, when kdump fails to create a vmcore file at the configured target location, the system reboots and the dump is lost in the process. To change this behavior, follow the procedure below.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.

Procedure

  1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.
  2. Replace the value with a desired action.

    failure_action poweroff

Additional resources

11.5. Configuration file for kdump

The configuration file for kdump kernel is /etc/sysconfig/kdump. This file controls the kdump kernel command line parameters.

For most configurations, use the default options. However, in some scenarios you might need to modify certain parameters to control the kdump kernel behavior. For example, modifying to append the kdump kernel command-line to obtain a detailed debugging output.

This section covers information on modifying the KDUMP_COMMANDLINE_REMOVE and KDUMP_COMMANDLINE_REMOVE options for kdump. For information on additional configuration options refer to Documentation/admin-guide/kernel-parameters.txt or the /etc/sysconfig/kdump file.

  • KDUMP_COMMANDLINE_REMOVE

    This option removes arguments from the current kdump command line. It removes parameters that may cause kdump errors or kdump kernel boot failures. These parameters may have been parsed from the previous KDUMP_COMMANDLINE process or inherited from the /proc/cmdline file. When this variable is not configured, it inherits all values from the /proc/cmdline file. Configuring this option also provides information that is helpful in debugging an issue.

    Example

    To remove certain arguments, add them to KDUMP_COMMANDLINE_REMOVE as follows:

    KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"
  • KDUMP_COMMANDLINE_APPEND

    This option appends arguments to the current command line. These arguments may have been parsed by the previous KDUMP_COMMANDLINE_REMOVE variable.

    For the kdump kernel, disabling certain modules such as mce, cgroup, numa, hest_disable can help prevent kernel errors. These modules may consume a significant portion of the kernel memory reserved for kdump or cause kdump kernel boot failures.

    Example

    To disable memory cgroups on the kdump kernel command line, run the command as follows:

    KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory"

Additional resources

  • Documentation/admin-guide/kernel-parameters.txt file
  • /etc/sysconfig/kdump file

11.6. Enabling and disabling the kdump service

To start the kdump service at boot time, follow the procedure below.

Prerequisites

  • Fulfilled kdump requirements for configurations and targets.
  • All configurations for installing kdump are set up according to your needs.

Procedure

  1. To enable the kdump service, use the following command:

    # systemctl enable kdump.service

    This enables the service for multi-user.target.

  2. To start the service in the current session, use the following command:

    # systemctl start kdump.service
  3. To stop the kdump service, type the following command:

    # systemctl stop kdump.service
  4. To disable the kdump service, execute the following command:

    # systemctl disable kdump.service
Warning

It is recommended to set kptr_restrict=1 as default. When kptr_restrict is set to (1) as default, the kdumpctl service loads the crash kernel even if Kernel Address Space Layout (KASLR) is enabled or not enabled.

Troubleshooting step

When kptr_restrict is not set to (1), and if KASLR is enabled, the contents of /proc/kore file are generated as all zeros. Consequently, the kdumpctl service fails to access the /proc/kcore and load the crash kernel.

To work around this problem, the kexec-kdump-howto.txt file displays a warning message, which specifies to keep the recommended setting as kptr_restrict=1.

To ensure that kdumpctl service loads the crash kernel, verify that:

  • Kernel kptr_restrict=1 in the sysctl.conf file.

Additional resources

11.7. Testing the kdump configuration

The following procedure describes how to test that the kernel dump process works and is valid before the machine enters production.

Warning

The commands below cause the kernel to crash. Use caution when following these steps, and never carelessly use them on active production system.

Procedure

  1. Reboot the system with kdump enabled.
  2. Make sure that kdump is running:

    ~]# systemctl is-active kdump
    active
  3. Force the Linux kernel to crash:

    echo 1 > /proc/sys/kernel/sysrq
    echo c > /proc/sysrq-trigger
    Warning

    The command above crashes the kernel and a reboot is required.

    Once booted again, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in /etc/kdump.conf (by default to /var/crash/).

    Note

    In addition to confirming the validity of the configuration, it is possible to use this action to record how long it takes for a crash dump to complete, while a representative load was running.

Additional resources

11.8. Using kexec to reboot the kernel

The kexec system call enables loading and booting into another kernel from the currently running kernel, thus performing a function of a boot loader from within the kernel.

The kexec utility loads the kernel and the initramfs image for the kexec system call to boot into another kernel.

The following procedure describes how to manually invoke the kexec system call when using the kexec utility to reboot into another kernel.

Procedure

  1. Execute the kexec utility:

    # kexec -l /boot/vmlinuz-3.10.0-1040.el7.x86_64 --initrd=/boot/initramfs-3.10.0-1040.el7.x86_64.img --reuse-cmdline

    The command manually loads the kernel and the initramfs image for the kexec system call.

  2. Reboot the system:

    # reboot

    The command detects the kernel, shuts down all services and then calls the kexec system call to reboot into the kernel you provided in the previous step.

Warning

When you use the kexec -e command to reboot the kernel, the system does not go through the standard shutdown sequence before starting the next kernel, which may cause data loss or an unresponsive system.

11.9. Preventing kernel drivers from loading for kdump

This section explains how to prevent the kdump kernel from loading certain kernel drivers using the `/etc/sysconfig/kdump configuration file. Appending the KDUMP_COMMANDLINE_APPEND= variable in /etc/sysconfig/kdump file, prevents the kdump initramfs from loading the specified kernel module. This helps to prevent oom killer or other crash kernel failures.

You can append the KDUMP_COMMANDLINE_APPEND= variable using one of the following configuration options:

  • rd.driver.blacklist=<modules>
  • modprobe.blacklist=<modules>

Procedure

  1. Select a kernel module that you intend to block from loading.

    $ lsmod
    Module                  Size  Used by
    fuse                  126976  3
    xt_CHECKSUM            16384  1
    ipt_MASQUERADE         16384  1
    uinput                 20480  1
    xt_conntrack           16384  1

    The lsmod command displays a list of modules that are loaded to the currently running kernel.

  2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file.

    KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

    Also,consider the following example using the modprobe.blacklist=<modules> configuration option.

    KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"
  3. Restart the kdump service.

    $ systemctl restart kdump

Additional resources

  • dracut.cmdline manual page

11.10. Running kdump on systems with encrypted disk

When running a LUKS encrypted partition, systems require certain amount of available memory. If the system has less than the required amount of available memory, the cryptsetup utility fails to mount the partition. As a result, capturing the vmcore file to an encrypted target location fails in the second kernel (capture kernel).

Using the kdumpctl estimate command, you can estimate the amount of memory required for kdump. kdumpctl estimate prints the Recommended crashkernel value, which is the recommended memory size required for kdump.

The Recommended crashkernel value is calculated based on the current kernel size, kernel module, initramfs, and the LUKS encrypted target memory requirement.

In case you are using the custom crashkernel= option, kdumpctl estimate prints the LUKS required size value, which is the memory size required for LUKS encrypted target.

Procedure

  1. Run the following command to print the estimate crashkernel= value:

    # kdumpctl estimate
    
    Encrypted kdump target requires extra memory, assuming using the keyslot  with minimum memory requirement
       Reserved crashkernel:    256M
       Recommended crashkernel: 652M
    
       Kernel image size:   47M
       Kernel modules size: 8M
       Initramfs size:      20M
       Runtime reservation: 64M
       LUKS required size:  512M
       Large modules: <none>
       WARNING: Current crashkernel size is lower than recommended size 652M.
  2. Configure the amount of required memory by increasing the crashkernel= value to the desired value.
  3. Reboot the system.
Note

If the kdump service still fails to save the dump file to the encrypted target, increase the crashkernel= value as required.