Chapter 10. Configuring kdump on the command line
The memory for kdump
is reserved during the system boot. The memory size is configured in the system’s Grand Unified Bootloader (GRUB) configuration file. The memory size depends on the crashkernel=
value specified in the configuration file and the size of the system’s physical memory.
10.1. Configuring kdump memory usage
The kexec-tools
package maintains the default crashkernel
memory reservation values. The kdump
service uses the default value to reserve the crashkernel
memory for each kernel.
The automatic memory allocation for kdump
varies based on the system hardware architecture and available memory size. For example, on the 64-bit ARM architecture, the crashkernel
default parameters work only when the available memory is more than 1 GB. kexec-tools
, by default, configures following memory reserve on the 64-bit ARM architecture:
crashkernel=1G-4G:256M,4G-64G:320M,64G:576M
The memory requirement of the crash kernel may vary depending on the hardware and machine specifications. If the default crashkernel
value does not work on your system, you can run the kdumpctl estimate
command and query a rough estimate value without triggering a crash. The estimated crashkernel
value may not be accurate and can serve as a reference to set an appropriate crashkernel
value.
The crashkernel=auto
option in the boot command line is no longer supported on RHEL 9 and later releases.
Prerequisites
- Root privileges
-
Fulfilled
kdump
requirements for configurations and targets. For details, see Supported kdump configurations and targets -
On IBM Z systems, ensure the
zipl
utility is installed.
Procedure
Configure the default value for
crashkernel
:# kdumpctl reset-crashkernel --kernel=ALL
(Optional) To use a custom
crashkernel
value:Configure the required memory reserve:
crashkernel=1G-4G:192M,4G-:256M
The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for
kdump
instead.(Optional) Offset the reserved memory:
Some systems require to reserve memory with a certain fixed offset since crash kernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:
crashkernel=192M@16M
The example above reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely,
kdump
offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example,crashkernel=1G-4G:192M,2G-64G:256M@16M
.
Update the bootloader configuration:
# grubby --update-kernel ALL --args "crashkernel=<CUSTOM-VALUE>”
On IBM Z systems that use the zIPL bootloader, the command adds a new kernel parameter to each
/boot/loader/entries/<ENTRY>.conf
file.On IBM Z systems, to update the boot menu, execute the
zipl
command with no options specified:# zipl
Reboot for changes to take effect:
# reboot
Verification
Activate the
sysrq
key to boot into thekdump
kernel:# echo 1 > /proc/sys/kernel/sysrq # echo c > /proc/sysrq-trigger
This forces the Linux kernel to crash and copy the
address-YYYY-MM-DD-HH:MM:SS/vmcore
file to the target location specified in the configuration file.Verify that the
vmcore
file is dumped in the target as specified in the/etc/kdump.conf
file:$ ls /var/crash/127.0.0.1-2022-01-18-0 /var/crash/127.0.0.1-2022-01-18-05:23:10': kexec-dmesg.log vmcore vmcore-dmesg.txt
In this example, the kernel saves the
vmcore
in the default target directory,/var/crash/
.
10.2. Configuring the kdump target
The crash dump is usually stored as a file in a local file system, written directly to a device. Alternatively, you can set up for the crash dump to be sent over a network using the NFS
or SSH
protocols. Only one of these options to preserve a crash dump file can be set at a time. The default behavior is to store it in the /var/crash/
directory of the local file system.
Prerequisites
-
Root
permissions. -
Fulfilled requirements for
kdump
configurations and targets. For details, see Supported kdump configurations and targets.
Procedure
To store the crash dump file in
/var/crash/
directory of the local file system, edit the/etc/kdump.conf
file and specify the path:path /var/crash
The option
path /var/crash
represents the path to the file system in whichkdump
saves the crash dump file.Note-
When you specify a dump target in the
/etc/kdump.conf
file, then the path is relative to the specified dump target. -
When you do not specify a dump target in the
/etc/kdump.conf
file, then the path represents the absolute path from the root directory.
Depending on what is mounted in the current system, the dump target and the adjusted dump path are taken automatically.
Example 10.1. The
kdump
target configuration# grep -v ^# /etc/kdump.conf | grep -v ^$ ext4 /dev/mapper/vg00-varcrashvol path /var/crash core_collector makedumpfile -c --message-level 1 -d 31
Here, the dump target is specified (
ext4 /dev/mapper/vg00-varcrashvol
), and thus mounted at/var/crash
. Thepath
option is also set to/var/crash
, so thekdump
saves thevmcore
file in the/var/crash/var/crash
directory.-
When you specify a dump target in the
To change the local directory in which the crash dump is to be saved, as
root
, edit the/etc/kdump.conf
configuration file:-
Remove the hash sign ("#") from the beginning of the
#path /var/crash
line. Replace the value with the intended directory path. For example:
path /usr/local/cores
ImportantIn RHEL 9, the directory defined as the kdump target using the
path
directive must exist when thekdump
systemd
service is started - otherwise the service fails.
-
Remove the hash sign ("#") from the beginning of the
To write the file to a different partition, edit the
/etc/kdump.conf
configuration file:Remove the hash sign ("#") from the beginning of the
#ext4
line, depending on your choice.-
device name (the
#ext4 /dev/vg/lv_kdump
line) -
file system label (the
#ext4 LABEL=/boot
line) -
UUID (the
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
line)
-
device name (the
Change the file system type as well as the device name, label or UUID to the desired values. For example:
ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
- NOTE
The correct syntax for specifying UUID values is both
UUID="correct-uuid"
andUUID=correct-uuid
.ImportantIt is recommended to specify storage devices using a
LABEL=
orUUID=
. Disk device names such as/dev/sda3
are not guaranteed to be consistent across reboot.ImportantWhen dumping to Direct Access Storage Device (DASD) on IBM Z hardware, it is essential that the dump devices are correctly specified in
/etc/dasd.conf
before proceeding.
To write the crash dump directly to a device, edit the
/etc/kdump.conf
configuration file:-
Remove the hash sign ("#") from the beginning of the
#raw /dev/vg/lv_kdump
line. Replace the value with the intended device name. For example:
raw /dev/sdb1
-
Remove the hash sign ("#") from the beginning of the
To store the crash dump to a remote machine using the
NFS
protocol:-
Remove the hash sign ("#") from the beginning of the
#nfs my.server.com:/export/tmp
line. Replace the value with a valid hostname and directory path. For example:
nfs penguin.example.com:/export/cores
-
Remove the hash sign ("#") from the beginning of the
To store the crash dump to a remote machine using the
SSH
protocol:-
Remove the hash sign ("#") from the beginning of the
#ssh user@my.server.com
line. - Replace the value with a valid username and hostname.
Include your
SSH
key in the configuration.-
Remove the hash sign from the beginning of the
#sshkey /root/.ssh/kdump_id_rsa
line. Change the value to the location of a key valid on the server you are trying to dump to. For example:
ssh john@penguin.example.com sshkey /root/.ssh/mykey
-
Remove the hash sign from the beginning of the
-
Remove the hash sign ("#") from the beginning of the
10.3. Configuring the core collector
The kdump
service uses the core_collector
program to capture the vmcore
image. In RHEL, the makedumpfile
utility is the default core collector.
makedumpfile
is a dump program that helps to copy only the required pages using various dump levels and compress the size of a dump file.
makedumpfile is a dump program that helps to copy only necessary pages using various dump levels and compress the size of a dump file.
Using makedumpfile
, you can create a small size dump file either by compressing dump data or by excluding pages or both. It needs the first kernel debug information to distinguish the not necessary pages by analyzing how the first kernel uses the memory.
Syntax
core_collector makedumpfile -z -d 31 --message-level 1
Options
-
-c
,-l
,-z
, or-p
: specifies the compress dump file format by each page when you use one of these options:-c
forzlib
,-l
forlzo
,-z
forzstd
, or-p
forsnappy
. -
-d
(dump_level)
: excludes pages so that they do not get copied to the dump file. --message-level
: specifies the message types.Using
--message-level
, you can restrict the outputs to print. For example, specifying 7 as the message level prints common messages and error messages. The maximum value for--message_level
is 31.
Prerequisites
-
Fulfilled
kdump
requirements for configurations and targets.
Procedure
As
root
user, edit the/etc/kdump.conf
configuration file to remove the hash sign ("#") from the beginning of the following command:core_collector makedumpfile -z -d 31 --message-level 1
To enable dump file compression, specify one of the
makedumpfile
options:core_collector makedumpfile -z -d 31 --message-level 1
where,
-
-z
specifies thedump
compressed file format. -
-d
specifies dump level as 31. -
--message-level
specifies message level as 1.
-
Also, consider the following example that uses -l
:
To compress a dump file using
-l
:core_collector makedumpfile -l -d 31 --message-level 1
Additional resources
-
makedumpfile(8)
manual page
10.4. Configuring the kdump default failure responses
By default, when kdump
fails to create a crash dump file at the configured target location, the system reboots and the dump is lost in the process. To change this behavior, follow the procedure below.
Prerequisites
- Root permissions.
-
Fulfilled requirements for
kdump
configurations and targets.
Procedure
-
As
root
, remove the hash sign ("#") from the beginning of the#failure_action
line in the/etc/kdump.conf
configuration file. Replace the value with a desired action.
failure_action poweroff
10.5. Configuration file for kdump
The configuration file for kdump
kernel is /etc/sysconfig/kdump
. This file controls the kdump
kernel command line parameters.
For most configurations, use the default options. However, in some scenarios you might need to modify certain parameters to control the kdump
kernel behavior. For example, modifying to append the kdump
kernel command-line to obtain a detailed debugging output.
This section covers information on modifying the KDUMP_COMMANDLINE_REMOVE
and KDUMP_COMMANDLINE_APPEND
options for kdump
. For information on additional configuration options refer to Documentation/admin-guide/kernel-parameters.txt
or the /etc/sysconfig/kdump
file.
KDUMP_COMMANDLINE_REMOVE
This option removes arguments from the current
kdump
command line. It removes parameters that may causekdump
errors orkdump
kernel boot failures. These parameters may have been parsed from the previousKDUMP_COMMANDLINE
process or inherited from the/proc/cmdline
file. When this variable is not configured, it inherits all values from the/proc/cmdline
file. Configuring this option also provides information that is helpful in debugging an issue.Example
To remove certain arguments, add them to
KDUMP_COMMANDLINE_REMOVE
as follows:# KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"
KDUMP_COMMANDLINE_APPEND
This option appends arguments to the current command line. These arguments may have been parsed by the previous
KDUMP_COMMANDLINE_REMOVE
variable.For the
kdump
kernel, disabling certain modules such asmce
,cgroup
,numa
,hest_disable
can help prevent kernel errors. These modules may consume a significant portion of the kernel memory reserved forkdump
or causekdump
kernel boot failures.Example
To disable memory
cgroups
on thekdump
kernel command line, run the command as follows:# KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory"
Additional resources
-
Documentation/admin-guide/kernel-parameters.txt
file -
/etc/sysconfig/kdump
file
10.6. Enabling and disabling the kdump service
To start the kdump
service at boot time, follow the procedure below.
Prerequisites
-
Fulfilled
kdump
requirements for configurations and targets. -
All configurations for installing
kdump
are set up according to your needs.
Procedure
To enable the
kdump
service, use the following command:# systemctl enable kdump.service
This enables the service for
multi-user.target
.To start the service in the current session, use the following command:
# systemctl start kdump.service
To stop the
kdump
service, type the following command:# systemctl stop kdump.service
To disable the
kdump
service, execute the following command:# systemctl disable kdump.service
It is recommended to set kptr_restrict=1
as default. When kptr_restrict
is set to (1) as default, the kdumpctl
service loads the crash kernel even if Kernel Address Space Layout (KASLR) is enabled or not enabled.
Troubleshooting step
When kptr_restrict
is not set to (1), and if KASLR is enabled, the contents of /proc/kore
file are generated as all zeros. Consequently, the kdumpctl
service fails to access the /proc/kcore
and load the crash kernel.
To work around this problem, the kexec-kdump-howto.txt
file displays a warning message, which specifies to keep the recommended setting as kptr_restrict=1
.
To ensure that kdumpctl
service loads the crash kernel, verify that:
-
Kernel
kptr_restrict=1
in thesysctl.conf
file.
10.7. Testing the kdump configuration
You can test that the crash dump process works and is valid before the machine enters production.
The commands below cause the kernel to crash. Use caution when following these steps, and never carelessly use them on active production system.
Procedure
-
Reboot the system with
kdump
enabled. Make sure that
kdump
is running:# systemctl is-active kdump active
Force the Linux kernel to crash:
echo 1 > /proc/sys/kernel/sysrq echo c > /proc/sysrq-trigger
WarningThe command above crashes the kernel, and a reboot is required.
Once booted again, the
address-YYYY-MM-DD-HH:MM:SS/vmcore
file is created at the location you have specified in the/etc/kdump.conf
file (by default to/var/crash/
).NoteThis action confirms the validity of the configuration. Also it is possible to use this action to record how long it takes for a crash dump to complete with a representative work-load.
Additional resources
10.8. Preventing kernel drivers from loading for kdump
This section explains how to prevent the capture kernel from loading certain kernel drivers using the /etc/sysconfig/kdump
configuration file. You can prevent the kdump
initramfs from loading the specified kernel module. To achieve this, you need to put the KDUMP_COMMANDLINE_APPEND=
variable in the /etc/sysconfig/kdump
file. This helps to prevent the out-of-memory (oom) killer or other crash kernel failures.
You can append the KDUMP_COMMANDLINE_APPEND=
variable using one of the following configuration options:
-
rd.driver.blacklist=<modules>
-
modprobe.blacklist=<modules>
Procedure
Select a kernel module that you intend to block from loading.
$ lsmod Module Size Used by fuse 126976 3 xt_CHECKSUM 16384 1 ipt_MASQUERADE 16384 1 uinput 20480 1 xt_conntrack 16384 1
The
lsmod
command displays a list of modules that are loaded to the currently running kernel.Update the
KDUMP_COMMANDLINE_APPEND=
variable in the/etc/sysconfig/kdump
file.# KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"
Also,consider the following example using the
modprobe.blacklist=<modules>
configuration option.# KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"
Restart the
kdump
service.# systemctl restart kdump
Additional resources
-
dracut.cmdline
manual page
10.9. Running kdump on systems with encrypted disk
When you run a Linux Unified Key Setup (LUKS) encrypted partition, the system require certain amount of available memory. If the system has less than the required amount of available memory, the systemd-cryptsetup
service fails to mount the partition. As a result, capturing the vmcore
file to an encrypted target location fails in the second kernel (capture kernel).
The kdumpctl estimate
command helps you estimate the amount of memory you need for kdump
. It prints the recommended crashkernel
value, which is the most suitable memory size required for kdump
.
The recommended crashkernel
value is calculated based on the current kernel size, kernel modules, initramfs
, and the LUKS encrypted target memory requirement.
In case you use the custom crashkernel
option, kdumpctl estimate
prints the LUKS required size
value. The value is the memory size required for LUKS encrypted target.
Procedure
Print the estimate
crashkernel
value:# kdumpctl estimate Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement Reserved crashkernel: 256M Recommended crashkernel: 652M Kernel image size: 47M Kernel modules size: 8M Initramfs size: 20M Runtime reservation: 64M LUKS required size: 512M Large modules: none WARNING: Current crashkernel size is lower than recommended size 652M.
Configure the amount of required memory by increasing
crashkernel
to the desired value.# grubby –args=”crashkernel=652M” --update-kernel=ALL
Reboot for changes to take effect.
# reboot
If the kdump
service still fails to save the dump file to the encrypted target, increase the crashkernel
value gradually to configure an appropriate amount of memory.