Menu Close
Chapter 11. Configuring kdump on the command line
The memory for kdump
is reserved during the system boot. The memory size is configured in the system’s Grand Unified Bootloader (GRUB) 2 configuration file. The memory size depends on the crashkernel=
value specified in the configuration file and the size of the system’s physical memory.
11.1. Configuring kdump memory usage
The kexec-tools
package maintains the default crashkernel
memory reservation values. The kdump
service uses the default value to reserve the crashkernel
memory for each kernel. This default value can also form the reference base value to estimate the required memory size when setting crashkernel
value manually. The minimum size of the crash kernel may vary depending on the hardware and machine specifications.
The automatic memory allocation for kdump
also varies based on the system hardware architecture and available memory size. For example, on AMD and Intel 64-bit architectures, the crashkernel
default parameters work only when the available memory is more than 1GB. By default, kexec-tools
configures the following memory reserves on AMD64 and Intel 64-bit architectures:
crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
You can also run kdumpctl estimate
to query a rough estimate value without triggering a crash. The estimated crashkernel
value may not be accurate and can serve as a reference to set an appropriate crashkernel
value.
The crashkernel=auto
option in the boot command line is no longer supported on RHEL 9 and later releases.
Prerequisites
- Root privileges
-
Fulfilled
kdump
requirements for configurations and targets.
Procedure
Configure default value for
crashkernel
:# kdumpctl reset-crashkernel --kernel=ALL
To use a custom crashkernel value:
Configure the required memory reserve.
# crashkernel=192M
The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4GB. If the total amount of memory is more than 4GB, 256MB is reserved for
kdump
instead.(Optional) Offset the reserved memory:
Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:
# crashkernel=192M@16M
The example above reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely,
kdump
offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example,crashkernel=1G-4G:192M,2G-64G:256M@16M
.
Update the bootloader configuration:
# grubby --update-kernel ALL --args "crashkernel=<CUSTOM-VALUE>”
Reboot for changes to take effect:
# reboot
Verification
Activate the
sysrq
key to boot into thekdump
kernel:# echo 1 > /proc/sys/kernel/sysrq # echo c > /proc/sysrq-trigger
This forces the Linux kernel to crash and copy the
address-YYYY-MM-DD-HH:MM:SS/vmcore
file to the target location specified in the configuration file.Verify that the
vmcore
file is dumped in the target as specified in the/etc/kdump.conf
file.$ ls /var/crash/127.0.0.1-2022-01-18-0 /var/crash/127.0.0.1-2022-01-18-05:23:10': kexec-dmesg.log vmcore vmcore-dmesg.txt
In this example, the kernel saves the
vmcore
in the default target directory,/var/crash/
.
11.2. Configuring the kdump target
When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS
(Network File System) or SSH
(Secure Shell) protocol. Only one of these options can be set at a time, and the default behavior is to store the vmcore file in the /var/crash/
directory of the local file system.
Prerequisites
-
Fulfilled
kdump
requirements for configurations and targets.
Procedure
To store the
vmcore
file in/var/crash/
directory of the local file system, edit the/etc/kdump.conf
file and specify the path:# path /var/crash
The option
path /var/crash
represents the path to the file system in whichkdump
saves thevmcore
file. When you specify a dump target in the/etc/kdump.conf
file, then thepath
is relative to the specified dump target.If you do not specify a dump target in the
/etc/kdump.conf
file, then thepath
represents the absolute path from the root directory. Depending on what is mounted in the current system, the dump target and the adjusted dump path are taken automatically.
kdump
saves the vmcore
file in /var/crash/var/crash
directory, when the dump target is mounted at /var/crash
and the option path
is also set as /var/crash
in the /etc/kdump.conf
file. For example, in the following instance, the ext4
file system is already mounted at /var/crash
and the path
are set as /var/crash
:
grep -v ^# etc/kdump.conf | grep -v ^$ ext4 /dev/mapper/vg00-varcrashvol path /var/crash core_collector makedumpfile -c --message-level 1 -d 31
This results in the /var/crash/var/crash
path. To solve this problem, use the option path /
instead of path /var/crash
To change the local directory in which the core dump is to be saved, as
root
, edit the/etc/kdump.conf
configuration file as described below.-
Remove the hash sign ("#") from the beginning of the
#path /var/crash
line. Replace the value with the intended directory path. For example:
# path /usr/local/cores
ImportantIn RHEL 9, the directory defined as the kdump target using the
path
directive must exist when thekdump
systemd service is started - otherwise the service fails. This behavior is different from earlier releases of RHEL, where the directory was being created automatically if it did not exist when starting the service.
-
Remove the hash sign ("#") from the beginning of the
To write the file to a different partition, as
root
, edit the/etc/kdump.conf
configuration file as described below.Remove the hash sign ("#") from the beginning of the
#ext4
line, depending on your choice.-
device name (the
#ext4 /dev/vg/lv_kdump
line) -
file system label (the
#ext4 LABEL=/boot
line) -
UUID (the
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
line)
-
device name (the
Change the file system type as well as the device name, label or UUID to the desired values. For example:
# ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
ImportantIt is recommended to specify storage devices using a
LABEL=
orUUID=
. Disk device names such as/dev/sda3
are not guaranteed to be consistent across reboot.ImportantWhen dumping to Direct Access Storage Device (DASD) on IBM Z hardware, it is essential that the dump devices are correctly specified in
/etc/dasd.conf
before proceeding.
To write the dump directly to a device:
-
Remove the hash sign ("#") from the beginning of the
#raw /dev/vg/lv_kdump
line. Replace the value with the intended device name. For example:
# raw /dev/sdb1
-
Remove the hash sign ("#") from the beginning of the
To store the dump to a remote machine using the
NFS
protocol:-
Remove the hash sign ("#") from the beginning of the
#nfs my.server.com:/export/tmp
line. Replace the value with a valid hostname and directory path. For example:
# nfs penguin.example.com:/export/cores
-
Remove the hash sign ("#") from the beginning of the
To store the dump to a remote machine using the
SSH
protocol:-
Remove the hash sign ("#") from the beginning of the
#ssh user@my.server.com
line. - Replace the value with a valid username and hostname.
Include your
SSH
key in the configuration.-
Remove the hash sign from the beginning of the
#sshkey /root/.ssh/kdump_id_rsa
line. Change the value to the location of a key valid on the server you are trying to dump to. For example:
# ssh john@penguin.example.com sshkey /root/.ssh/mykey
-
Remove the hash sign from the beginning of the
-
Remove the hash sign ("#") from the beginning of the
11.3. Configuring the core collector
The kdump
service uses the core_collector
program to capture the vmcore
image. In RHEL, the makedumpfile
utility is the default core collector.
makedumpfile
is a dump program that helps to copy only the required pages using various dump levels and compress the size of a dump file.
makedumpfile is a dump program that helps to copy only necessary pages using various dump levels and compress the size of a dump file.
Using makedumpfile
, you can create a small size dump file either by compressing dump data or by excluding pages or both. It needs the first kernel debug information to distinguish the not necessary pages by analyzing how the first kernel uses the memory.
Syntax
core_collector makedumpfile -z -d 31 --message-level 1
Options
-
-c
,-l
,-z
, or-p
: specifies the compress dump file format by each page when you use one of these options:-c
forzlib
,-l
forlzo
,-z
forzstd
, or-p
forsnappy
. -
-d
(dump_level)
: excludes pages so that they do not get copied to the dump file. --message-level
: specifies the message types.Using
--message-level
, you can restrict the outputs to print. For example, specifying 7 as the message level prints common messages and error messages. The maximum value for--message_level
is 31.
Prerequisites
-
Fulfilled
kdump
requirements for configurations and targets.
Procedure
As
root
user, edit the/etc/kdump.conf
configuration file to remove the hash sign ("#") from the beginning of the following command:#core_collector makedumpfile -z -d 31 --message-level 1
To enable dump file compression, specify one of the
makedumpfile
options:core_collector makedumpfile -z -d 31 --message-level 1
where,
-
-z
specifies thedump
compressed file format. -
-d
specifies dump level as 31. -
--message-level
specifies message level as 1.
-
Also, consider the following example that uses -l
:
To compress a dump file using
-l
:core_collector makedumpfile -l -d 31 --message-level 1
Additional resources
-
makedumpfile(8)
manual page
11.4. Configuring the kdump default failure responses
By default, when kdump
fails to create a crash dump file at the configured target location, the system reboots and the dump is lost in the process. To change this behavior, follow the procedure below.
Prerequisites
- Root permissions.
-
Fulfilled requirements for
kdump
configurations and targets.
Procedure
-
As
root
, remove the hash sign ("#") from the beginning of the#failure_action
line in the/etc/kdump.conf
configuration file. Replace the value with a desired action.
# failure_action poweroff
11.5. Configuration file for kdump
The configuration file for kdump
kernel is /etc/sysconfig/kdump
. This file controls the kdump
kernel command line parameters.
For most configurations, use the default options. However, in some scenarios you might need to modify certain parameters to control the kdump
kernel behavior. For example, modifying to append the kdump
kernel command-line to obtain a detailed debugging output.
This section covers information on modifying the KDUMP_COMMANDLINE_REMOVE
and KDUMP_COMMANDLINE_APPEND
options for kdump
. For information on additional configuration options refer to Documentation/admin-guide/kernel-parameters.txt
or the /etc/sysconfig/kdump
file.
KDUMP_COMMANDLINE_REMOVE
This option removes arguments from the current
kdump
command line. It removes parameters that may causekdump
errors orkdump
kernel boot failures. These parameters may have been parsed from the previousKDUMP_COMMANDLINE
process or inherited from the/proc/cmdline
file. When this variable is not configured, it inherits all values from the/proc/cmdline
file. Configuring this option also provides information that is helpful in debugging an issue.Example
To remove certain arguments, add them to
KDUMP_COMMANDLINE_REMOVE
as follows:# KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb"
KDUMP_COMMANDLINE_APPEND
This option appends arguments to the current command line. These arguments may have been parsed by the previous
KDUMP_COMMANDLINE_REMOVE
variable.For the
kdump
kernel, disabling certain modules such asmce
,cgroup
,numa
,hest_disable
can help prevent kernel errors. These modules may consume a significant portion of the kernel memory reserved forkdump
or causekdump
kernel boot failures.Example
To disable memory
cgroups
on thekdump
kernel command line, run the command as follows:# KDUMP_COMMANDLINE_APPEND="cgroup_disable=memory"
Additional resources
-
Documentation/admin-guide/kernel-parameters.txt
file -
/etc/sysconfig/kdump
file
11.6. Enabling and disabling the kdump service
To start the kdump
service at boot time, follow the procedure below.
Prerequisites
-
Fulfilled
kdump
requirements for configurations and targets. -
All configurations for installing
kdump
are set up according to your needs.
Procedure
To enable the
kdump
service, use the following command:# systemctl enable kdump.service
This enables the service for
multi-user.target
.To start the service in the current session, use the following command:
# systemctl start kdump.service
To stop the
kdump
service, type the following command:# systemctl stop kdump.service
To disable the
kdump
service, execute the following command:# systemctl disable kdump.service
It is recommended to set kptr_restrict=1
as default. When kptr_restrict
is set to (1) as default, the kdumpctl
service loads the crash kernel even if Kernel Address Space Layout (KASLR) is enabled or not enabled.
Troubleshooting step
When kptr_restrict
is not set to (1), and if KASLR is enabled, the contents of /proc/kore
file are generated as all zeros. Consequently, the kdumpctl
service fails to access the /proc/kcore
and load the crash kernel.
To work around this problem, the kexec-kdump-howto.txt
file displays a warning message, which specifies to keep the recommended setting as kptr_restrict=1
.
To ensure that kdumpctl
service loads the crash kernel, verify that:
-
Kernel
kptr_restrict=1
in thesysctl.conf
file.
11.7. Testing the kdump configuration
You can test that the crash dump process works and is valid before the machine enters production.
The commands below cause the kernel to crash. Use caution when following these steps, and never carelessly use them on active production system.
Procedure
-
Reboot the system with
kdump
enabled. Make sure that
kdump
is running:# systemctl is-active kdump active
Force the Linux kernel to crash:
# echo 1 > /proc/sys/kernel/sysrq # echo c > /proc/sysrq-trigger
WarningThe command above crashes the kernel, and a reboot is required.
Once booted again, the
address-YYYY-MM-DD-HH:MM:SS/vmcore
file is created at the location you have specified in the/etc/kdump.conf
file (by default to/var/crash/
).NoteThis action confirms the validity of the configuration. Also it is possible to use this action to record how long it takes for a crash dump to complete with a representative work-load.
11.8. Preventing kernel drivers from loading for kdump
This section explains how to prevent the capture kernel from loading certain kernel drivers using the /etc/sysconfig/kdump
configuration file. You can prevent the kdump
initramfs from loading the specified kernel module. To achieve this, you need to put the KDUMP_COMMANDLINE_APPEND=
variable in the /etc/sysconfig/kdump
file. This helps to prevent the out-of-memory (oom) killer or other crash kernel failures.
You can append the KDUMP_COMMANDLINE_APPEND=
variable using one of the following configuration options:
-
rd.driver.blacklist=<modules>
-
modprobe.blacklist=<modules>
Procedure
Select a kernel module that you intend to block from loading.
$ lsmod Module Size Used by fuse 126976 3 xt_CHECKSUM 16384 1 ipt_MASQUERADE 16384 1 uinput 20480 1 xt_conntrack 16384 1
The
lsmod
command displays a list of modules that are loaded to the currently running kernel.Update the
KDUMP_COMMANDLINE_APPEND=
variable in the/etc/sysconfig/kdump
file.# KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"
Also,consider the following example using the
modprobe.blacklist=<modules>
configuration option.# KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"
Restart the
kdump
service.# systemctl restart kdump
Additional resources
-
dracut.cmdline
manual page
11.9. Running kdump on systems with encrypted disk
When you run a Linux Unified Key Setup (LUKS) encrypted partition, the system require certain amount of available memory. If the system has less than the required amount of available memory, the systemd-cryptsetup
service fails to mount the partition. As a result, capturing the vmcore
file to an encrypted target location fails in the second kernel (capture kernel).
The kdumpctl estimate
command helps you estimate the amount of memory you need for kdump
. It prints the recommended crashkernel
value, which is the most suitable memory size required for kdump
.
The recommended crashkernel
value is calculated based on the current kernel size, kernel modules, initramfs
, and the LUKS encrypted target memory requirement.
In case you use the custom crashkernel
option, kdumpctl estimate
prints the LUKS required size
value. The value is the memory size required for LUKS encrypted target.
Procedure
Print the estimate
crashkernel
value:# kdumpctl estimate Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement Reserved crashkernel: 256M Recommended crashkernel: 652M Kernel image size: 47M Kernel modules size: 8M Initramfs size: 20M Runtime reservation: 64M LUKS required size: 512M Large modules: none WARNING: Current crashkernel size is lower than recommended size 652M.
Configure the amount of required memory by increasing
crashkernel
to the desired value.# grubby –args=”crashkernel=652M” --update-kernel=ALL
Reboot for changes to take effect.
# reboot
If the kdump
service still fails to save the dump file to the encrypted target, increase the crashkernel
value gradually to configure an appropriate amount of memory.