Menu Close

Chapter 19. Configuring kdump on the command line

The following sections explain how to plan and build your kdump environment.

19.1. Estimating the kdump size

When planning and building your kdump environment, it is important to know how much space the crash dump file requires.

The makedumpfile --mem-usage command estimates how much space the crash dump file requires. It generates a memory usage report. The report helps you determine the dump level and which pages are safe to be excluded.

Procedure

  • Execute the following command to generate a memory usage report:

    # makedumpfile --mem-usage /proc/kcore
    
    
    TYPE        PAGES    EXCLUDABLE    DESCRIPTION
    -------------------------------------------------------------
    ZERO          501635      yes        Pages filled with zero
    CACHE         51657       yes        Cache pages
    CACHE_PRIVATE 5442        yes        Cache pages + private
    USER          16301       yes        User process pages
    FREE          77738211    yes        Free pages
    KERN_DATA     1333192     no         Dumpable kernel data
Important

The makedumpfile --mem-usage command reports required memory in pages. This means that you must calculate the size of memory in use against the kernel page size.

19.2. Configuring kdump memory usage

The memory for kdump is reserved during the system boot. The memory size is configured in the system Grand Unified Bootloader (GRUB) 2 configuration file. The memory size depends on the value of the crashkernel= option specified in the configuration file and the size of the system physical memory.

The crashkernel= option can be defined in multiple ways. You can either specify the crashkernel= value or configure the auto option. The crashkernel=auto parameter reserves memory automatically, based on the total amount of physical memory in the system. When configured, the kernel will automatically reserve an appropriate amount of required memory for the capture kernel. This helps to prevent Out-of-Memory (OOM) errors.

Note

The automatic memory allocation for kdump varies based on system hardware architecture and available memory size.

If the system has less than the minimum memory threshold for automatic allocation, you can configure the amount of reserved memory manually.

Prerequisites

Procedure

  1. Edit the /etc/default/grub file.
  2. Set the crashkernel= option.

    For example, to reserve 128 MB of memory, use the following:

    crashkernel=128M

    Alternatively, you can set the amount of reserved memory to a variable depending on the total amount of installed memory. The syntax for memory reservation into a variable is crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

    crashkernel=512M-2G:64M,2G-:128M

    The above example reserves 64 MB of memory if the total amount of system memory is between 512 MB and 2 GB. If the total amount of memory is more than 2 GB, 128 MB is reserved.

    • Offset the reserved memory.

      Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

      crashkernel=128M@16M

      In the example above kdump reserves 128 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump offsets the reserved memory automatically. You can also use this syntax when setting a variable memory reservation. In that case, the offset is always specified last (for example, crashkernel=512M-2G:64M,2G-:128M@16M).

  3. Use the following command to update the GRUB2 configuration file:

    # grub2-mkconfig -o /boot/grub2/grub.cfg
Note

The alternative way to configure memory for kdump is to append the crashkernel=<SOME_VALUE> parameter to the kernelopts variable with the grub2-editenv command, which will update all of your boot entries. Or you can use the grubby utility to update one boot entry, more boot entries, or all of your boot entries.

19.3. Configuring the kdump target

The crash dump is usually stored as a file in a local file system, written directly to a device. Alternatively, you can set up for the crash dump to be sent over a network using the NFS or SSH protocols. Only one of these options to preserve a crash dump file can be set at a time. The default behavior is to store it in the /var/crash/ directory of the local file system.

Prerequisites

Procedure

  • To store the crash dump file in /var/crash/ directory of the local file system, edit the /etc/kdump.conf file and specify the path:

    path /var/crash

    The option path /var/crash represents the path to the file system in which kdump saves the crash dump file. When you specify a dump target in the /etc/kdump.conf file, then the path is relative to the specified dump target.

    If you do not specify a dump target in the /etc/kdump.conf file, then the path represents the absolute path from the root directory. Depending on what is mounted in the current system, the dump target and the adjusted dump path are taken automatically.

kdump saves the crash dump file in /var/crash/var/crash directory, when the dump target is mounted at /var/crash and the option path is also set as /var/crash in the /etc/kdump.conf file. For example, in the following instance, the ext4 file system is already mounted at /var/crash and the path are set as /var/crash:

grep -v ^# etc/kdump.conf | grep -v ^$
ext4 /dev/mapper/vg00-varcrashvol
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31

This results in the /var/crash/var/crash path. To solve this problem, use the option path / instead of path /var/crash

  • To change the local directory in which the crash dump is to be saved, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #path /var/crash line.
    2. Replace the value with the intended directory path. For example:

      path /usr/local/cores
      Important

      In RHEL 8, the directory defined as the kdump target using the path directive must exist when the kdump systemd service is started - otherwise the service fails. This behavior is different from earlier releases of RHEL, where the directory was being created automatically if it did not exist when starting the service.

  • To write the file to a different partition, as root, edit the /etc/kdump.conf configuration file as described below.

    1. Remove the hash sign ("#") from the beginning of the #ext4 line, depending on your choice.

      • device name (the #ext4 /dev/vg/lv_kdump line)
      • file system label (the #ext4 LABEL=/boot line)
      • UUID (the #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 line)
    2. Change the file system type as well as the device name, label or UUID to the desired values. For example:

      ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
      Important

      It is recommended to specify storage devices using a LABEL= or UUID=. Disk device names such as /dev/sda3 are not guaranteed to be consistent across reboot.

  • To write the crash dump directly to a device, edit the /etc/kdump.conf configuration file:

    1. Remove the hash sign ("#") from the beginning of the #raw /dev/vg/lv_kdump line.
    2. Replace the value with the intended device name. For example:

      raw /dev/sdb1
  • To store the crash dump to a remote machine using the NFS protocol, edit the /etc/kdump.conf configuration file:

    1. Remove the hash sign ("#") from the beginning of the #nfs my.server.com:/export/tmp line.
    2. Replace the value with a valid hostname and directory path. For example:

      nfs penguin.example.com:/export/cores
  • To store the crash dump to a remote machine using the SSH protocol, edit the /etc/kdump.conf configuration file:

    1. Remove the hash sign ("#") from the beginning of the #ssh user@my.server.com line.
    2. Replace the value with a valid username and hostname.
    3. Include your SSH key in the configuration.

      • Remove the hash sign from the beginning of the #sshkey /root/.ssh/kdump_id_rsa line.
      • Change the value to the location of a key valid on the server you are trying to dump to. For example:

        ssh john@penguin.example.com
        sshkey /root/.ssh/mykey

19.4. Configuring the kdump core collector

The kdump service uses a core_collector program to capture the crash dump image. In RHEL, the makedumpfile utility is the default core collector. It helps shrink the dump file by:

  • Compressing the size of a crash dump file and copying only necessary pages using various dump levels
  • Excluding unnecessary crash dump pages
  • Filtering the page types to be included in the crash dump.

Syntax

core_collector makedumpfile -l --message-level 1 -d 31

Options

  • -c, -l or -p: specify compress dump file format by each page using either, zlib for -c option, lzo for -l option or snappy for -p option.
  • -d (dump_level): excludes pages so that they are not copied to the dump file.
  • --message-level : specify the message types. You can restrict outputs printed by specifying message_level with this option. For example, specifying 7 as message_level prints common messages and error messages. The maximum value of message_level is 31

Prerequisites

Procedure

  1. As root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.
  2. To enable crash dump file compression, execute:
core_collector makedumpfile -l --message-level 1 -d 31

The -l option specifies the dump compressed file format. The -d option specifies dump level as 31. The --message-level option specifies message level as 1.

Also, consider following examples with the -c and -p options:

  • To compress a crash dump file using -c:
core_collector makedumpfile -c -d 31 --message-level 1
  • To compress a crash dump file using -p:
core_collector makedumpfile -p -d 31 --message-level 1

Additional resources

19.5. Configuring the kdump default failure responses

By default, when kdump fails to create a crash dump file at the configured target location, the system reboots and the dump is lost in the process. To change this behavior, follow the procedure below.

Prerequisites

Procedure

  1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.
  2. Replace the value with a desired action.

    failure_action poweroff

Additional resources

19.6. The kdump configuration file

The kdump configuration file, /etc/kdump.conf, contains options and commands for the kernel crash dump.

The first part of the file provides comments explaining the available options and commands. The second part of the file includes a default configuration. Options that are not in the default configuration are commented out using a hash mark at the start of each option. This makes it easy to modify the file correctly.

Note

The information here includes only some of the options that can be configured in this file.

19.7. kdump.conf configuration options

Memory to reserve

The crashkernel parameter defines the amount of memory reserved for the kernel crash dump. The following options are available:

  • An absolute value in megabytes

    For example: crashkernel=128M for 128 megabytes of reserved memory.

  • Some systems require that kdump memory is reserved with a fixed offset. This is because the crashkernel reservation is very early in the boot, and the system needs to reserve some memory for special usage. If an offset is configured, the reserved memory begins there.

    For example, crashkernel=128M@16M for 128 megabytes of reserved memory offset by 16 megabytes

  • Variable amounts. The amount of memory reserved is based on the amount of memory in the system.

    For example, crashkernel=512M-2G:64M,2G-:128M@16M for reserving 64 megabytes in a system with between 1/2 a megabyte and two gigabybtes of memory and 128 megabytes for systems with more than two gigabybtes of memory.

    Note

    You can combine variable amounts with offsets.

    For example, crashkernel=512M-2G:64M,2G-:128M@16M.

  • auto - Automatically allocates memory for the crash kernel dump based on the system hardware architecture and available memory size.

    If the system has less than the minimum memory threshold for automatic allocation, you can configure the amount of reserved memory manually.

Target

The location where the kernel crash dump will be saved. The following options are available:

  • raw - Defines a device to which the kernel crash dump will be sent. Use persistent device names for partition devices, such as /dev/vg/<devname>.
  • path - Defines the device, file system type, and the path to a directory on the local file system. You can specify the device using the device name (for example, /dev/vg/lv_kdump), file system label (for example, LABEL=/boot), or the UUID (for example, UUID=03138356-5e61-4ab3-b58e-27507ac41937).
  • nfs - Defines an NFS target with a hostname and directory path. For example, nfs penguin.example.com:/export/cores.
  • ssh - Defines an SSH target (for example, ssh john@penguin.example.com. The sshkey variable defines the location of the SSH key on the server.
Shrinking the dump file

The makedumpfile utility is a dump program that helps shrink the dump file using the following methods:

  • Compressing the size of a dump file using one of the following options:

    • -c - Compresses the file using the zlib utility
    • -l - Compresses the file using lzo utility
    • -p - Compresses the file using the snappy utility
  • Excluding unnecessary pages by using the -d option and specifying the pages to exclude. makedumpfile needs the first kernel debug information to understand how first kernel uses the memory. This helps it detect the pages that are needed for the dump.
  • Filtering the pages to be included in the dump using the --message-level option and specifying the page types to include by adding the following filtering options:

    • 1 - zero pages
    • 2 - cache pages
    • 4 - cache private pages
    • 8 - user pages
    • 16 - free pages

      For example, to specify that only cache pages, cache private pages, and user pages are included in the dump, specify --message-level 14 (2 + 4 + 8).

Note

The makedumpfile command supports removal of transparent huge pages and hugetlbfs pages from RHEL 7.3 and later. Consider both these types of pages user pages and remove them using the -8 option.

19.8. The kdump default failure response

When kdump fails to create a core dump, the default failure response of the operating system is to reboot. However, you can configure the kdump utility to perform a different operation in case it fails to save the core dump to the primary target.

Use the failure_action parameter to specify one of the following available default failure actions:

Option

Description

dump_to_rootfs

kdump tries to save the core dump to the root file system.

This option is especially useful in combination with a network target. If the network target is unreachable, this option configures kdump to save the core dump locally. The system reboots afterwards.

reboot

kdump reboots the system. The core dump is lost.

halt

kdump halts the system. The core dump is lost.

poweroff

kdump powers down the system. The core dump is lost.

shell

kdump opens a shell session from within the initramfs utility. This allows the user to record the core dump manually.

Additional resources

  • etc/kdump.conf

19.9. Testing the kdump configuration

You can test that the crash dump process works and is valid before the machine enters production.

Warning

The commands below cause the kernel to crash. Use caution when following these steps, and never carelessly use them on active production system.

Procedure

  1. Reboot the system with kdump enabled.
  2. Make sure that kdump is running:

    ~]# systemctl is-active kdump
    active
  3. Force the Linux kernel to crash:

    echo 1 > /proc/sys/kernel/sysrq
    echo c > /proc/sysrq-trigger
    Warning

    The command above crashes the kernel, and a reboot is required.

    Once booted again, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file (by default to /var/crash/).

    Note

    This action confirms the validity of the configuration. Also it is possible to use this action to record how long it takes for a crash dump to complete with a representative work-load.

Additional resources