Procedure

1. Edit the /etc/default/grub file using the root permissions.

2. Set the crashkernel= option to the required value.

   For example, to reserve 128 MB of memory, use the following:

   crashkernel=128M

   Alternatively, you can set the amount of reserved memory to a variable depending on the total amount of installed memory. The syntax for memory reservation into a variable is crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

   crashkernel=512M-2G:64M,2G-:128M

   The above example reserves 64 MB of memory if the total amount of system memory is 512 MB or higher and lower than 2 GB. If the total amount of memory is more than 2 GB, 128 MB is reserved for kdump instead.

   • Offset the reserved memory.

     Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

     crashkernel=128M@16M

     The example above means that kdump reserves 128 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump offsets the reserved memory automatically. This syntax can also be used when setting a variable memory reservation as described above; in this case, the offset is always specified last (for example, crashkernel=512M-2G:64M,2G-:128M@16M).

3. Use the following command to update the GRUB2 configuration file:

   # grub2-mkconfig -o /boot/grub2/grub.cfg

1. Remove the hash sign ("#") from the beginning of the #path /var/crash line.

2. Replace the value with the intended directory path. For example:

   path /usr/local/cores

   Important

   In Red Hat Enterprise Linux 8, the directory defined as the kdump target using the path directive must exist when the kdump systemd service is started - otherwise the service fails. This behavior is different from earlier releases of Red Hat Enterprise Linux, where the directory was being created automatically if it did not exist when starting the service.

1. Remove the hash sign ("#") from the beginning of the #ext4 line, depending on your choice.

   • device name (the #ext4 /dev/vg/lv_kdump line)
   • file system label (the #ext4 LABEL=/boot line)
   • UUID (the #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937 line)

2. Change the file system type as well as the device name, label or UUID to the desired values. For example:

   ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937

   Important

   It is recommended to specify storage devices using a LABEL= or UUID=. Disk device names such as /dev/sda3 are not guaranteed to be consistent across reboot.

   Important

   When dumping to Direct Access Storage Device (DASD) on IBM Z hardware, it is essential that the dump devices are correctly specified in /etc/dasd.conf before proceeding.

1. Remove the hash sign ("#") from the beginning of the #raw /dev/vg/lv_kdump line.

2. Replace the value with the intended device name. For example:

   raw /dev/sdb1

1. Remove the hash sign ("#") from the beginning of the #nfs my.server.com:/export/tmp line.

2. Replace the value with a valid hostname and directory path. For example:

   nfs penguin.example.com:/export/cores

1. Remove the hash sign ("#") from the beginning of the #ssh user@my.server.com line.
2. Replace the value with a valid username and hostname.

3. Include your SSH key in the configuration.

   • Remove the hash sign from the beginning of the #sshkey /root/.ssh/kdump_id_rsa line.

   • Change the value to the location of a key valid on the server you are trying to dump to. For example:

     ssh john@penguin.example.com
     sshkey /root/.ssh/mykey

Procedure

1. As root, edit the /etc/kdump.conf configuration file and remove the hash sign ("#") from the beginning of the #core_collector makedumpfile -l --message-level 1 -d 31.
2. To enable dump file compression, execute:

Procedure

1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a desired action as described in Section 10.5.5, “Supported default failure responses”. For example:

   failure_action poweroff

Procedure

1. To enable the kdump service, use the following command:

   # systemctl enable kdump.service

   This enables the service for multi-user.target.

2. To start the service in the current session, use the following command:

   # systemctl start kdump.service

3. To stop the kdump service, type the following command:

   # systemctl stop kdump.service

4. To disable the kdump service, execute the following command:

   # systemctl disable kdump.service

Procedure

1. Open the Kernel Dump tab and start the kdump service.
2. Configure the kdump memory usage through the command line.

3. Click the link next to the Crash dump location option.

   

4. Select the Local Filesystem option from the drop-down and specify the directory you want to save the dump in.

   

   • Alternatively, select the Remote over SSH option from the drop-down to send the vmcore to a remote machine using the SSH protocol.

     Fill the Server, ssh key, and Directory fields with the remote machine address, ssh key location, and a target directory.

   • Another choice is to select the Remote over NFS option from the drop-down and fill the Mount field to send the vmcore to a remote machine using the NFS protocol.

     Note

     Tick the Compression check box to reduce the size of the vmcore file.

5. Test your configuration by crashing the kernel.

   
   Warning

   This step disrupts execution of the kernel and results in a system crash and loss of data.

Procedure

1. Edit the `/etc/kdump.conf file and add the final_action parameter.

   final_action <reboot | halt | poweroff>

2. Restart the kdump service:

   kdumpctl restart

Procedure

1. Install and configure kdump as described in Chapter 7, "Installing and configuring kdump".

2. Add fadump=on to the GRUB_CMDLINE_LINUX line in /etc/default/grub file:

   GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel=auto
   rd.lvm.lv=rhel/root rhgb quiet fadump=on"

3. (Optional) If you want to specify reserved boot memory instead of using the defaults, configure crashkernel=xxM to GRUB_CMDLINE_LINUX in /etc/default/grub, where xx is the amount of the memory required in megabytes:

   GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel=xxM rd.lvm.lv=rhel/root rhgb quiet fadump=on"

   Important

   Red Hat recommends to test all boot configuration options before you execute them. If you observe Out of Memory (OOM) errors when booting from the crash kernel, increase the value specified in crashkernel= argument until the crash kernel can boot cleanly. Some trial and error may be required in this case.

Procedure

1. Add or edit the following lines in the /etc/sysctl.conf file to ensure that kdump starts as expected for sadump:

   kernel.panic=0
   kernel.unknown_nmi_panic=1

   Warning

   In particular, ensure that after kdump, the system does not reboot. If the system reboots after kdump has fails to save the vmcore file, then it is not possible to invoke the sadump.

2. Set the failure_action parameter in /etc/kdump.conf appropriately as halt or shell.

   failure_action shell

Procedure

1. Enable the relevant repositories:

   # subscription-manager repos --enable baseos repository

   # subscription-manager repos --enable appstream repository

   # subscription-manager repos --enable rhel-8-for-x86_64-baseos-debug-rpms

2. Install the crash package:

   # yum install crash

3. Install the kernel-debuginfo package:

   # yum install kernel-debuginfo

   The package corresponds to your running kernel and provides the data necessary for the dump analysis.

Procedure

1. To start the crash utility, two necessary parameters need to be passed to the command:

   • The debug-info (a decompressed vmlinuz image), for example /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux provided through a specific kernel-debuginfo package.

   • The actual vmcore file, for example /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     The resulting crash command then looks like this:

     # crash /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     Use the same <kernel> version that was captured by kdump.

     Example 10.1. Running the crash utility

     The following example shows analyzing a core dump created on October 6 2018 at 14:05 PM, using the 4.18.0-5.el8.x86_64 kernel.

     ...
     WARNING: kernel relocated [202MB]: patching 90160 gdb minimal_symbol values
     
           KERNEL: /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux
         DUMPFILE: /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore  [PARTIAL DUMP]
             CPUS: 2
             DATE: Sat Oct  6 14:05:16 2018
           UPTIME: 01:03:57
     LOAD AVERAGE: 0.00, 0.00, 0.00
            TASKS: 586
         NODENAME: localhost.localdomain
          RELEASE: 4.18.0-5.el8.x86_64
          VERSION: #1 SMP Wed Aug 29 11:51:55 UTC 2018
          MACHINE: x86_64  (2904 Mhz)
           MEMORY: 2.9 GB
            PANIC: "sysrq: SysRq : Trigger a crash"
              PID: 10635
          COMMAND: "bash"
             TASK: ffff8d6c84271800  [THREAD_INFO: ffff8d6c84271800]
              CPU: 1
            STATE: TASK_RUNNING (SYSRQ)
     
     crash>

2. To exit the interactive prompt and terminate crash, type exit or q.

   Example 10.2. Exiting the crash utility

   crash> exit
   ~]#

Procedure

1. Follow the Kernel Oops Analyzer link to access the tool.

2. To diagnose a kernel crash issue, upload a kernel oops log generated in vmcore.

   • Alternatively you can also diagnose a kernel crash issue by providing a text message or a vmcore-dmesg.txt as an input.

     
3. Click DETECT to compare the oops message based on information from the makedumpfile against known solutions.

Procedure

1. Verify that the kdump service is enabled and active:

   # systemctl is-enabled kdump.service && systemctl is-active kdump.service enabled active

   If kdump is not enabled and running see, Section 10.3.5, “Enabling and disabling the kdump service”.

2. Rebuild the initramfs image of the booting kernel with the early kdump functionality:

   dracut -f --add earlykdump

3. Add the rd.earlykdump kernel command line parameter:

   grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="rd.earlykdump"

4. Reboot the system to reflect the changes

   reboot

5. Optionally, verify that rd.earlykdump was successfully added and early kdump feature was enabled:

   # cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-187.el8.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet rd.earlykdump
   
   # journalctl -x | grep early-kdump
   Mar 20 15:44:41 redhat dracut-cmdline[304]: early-kdump is enabled.
   Mar 20 15:44:42 redhat dracut-cmdline[304]: kexec: loaded early-kdump kernel