3.2. Using kdump and kexec with the MRG Realtime kernel

kdump is a reliable kernel crash dumping mechanism because the crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. kdump uses a mechanism called kexec to boot into a second kernel whenever the system crashes. This second kernel, often called the crash kernel, boots with very little memory and captures the dump image.
If kdump is enabled on your system, the standard boot kernel will reserve a small section of system RAM and load the kdump kernel into the reserved space. When a kernel panic or other fatal error occurs, kexec is used to boot into the kdump kernel without going through BIOS. The system reboots to the kdump kernel that is confined to the memory space reserved by the standard boot kernel, and this kernel writes a copy or image of the system memory to the storage mechanism defined in the configuration files. Because kexec does not go through the BIOS, the memory of the original boot is retained, and the crash dump is much more detailed. Once this is done, the kernel reboots, which resets the machine and brings the boot kernel back up.

Note

Earlier versions of MRG Realtime used the stock Red Hat Enterprise Linux kernels as the kdump kernel. The Realtime kernel is the preferred kdump crash kernel on MRG 2.1 and later.
There are three required procedures for enabling kdump under Red Hat Enterprise Linux 6. First, ensure that the required RPM packages are installed on the system. Second, create the minimum configuration and modifies the grub command line using the rt-setup-kdump tool. Third, use a graphical system configuration tool called system-config-kdump to create and enable a detailed kdump configuration.
  1. Installing required kdump packages

    1. The rt-setup-kdump tool is part of the rt-setup package, which can be installed using yum:
      # yum install rt-setup
      
    2. Check that you have installed the latest versions of the kexec-tools and system-config-kdump packages available to your distribution.
      # rpm -q kexec-tools system-config-kdump
      kexec-tools-<version.number>.el6.arch
      system-config-kdump-<version.number>.el6.arch
  2. Creating a basic kdump kernel with rt-setup-kdump

    1. Run the rt-setup-kdump tool by invoking it at the shell prompt as the root user.
      # rt-setup-kdump --grub
      
      The --grub parameter adds the necessary changes to all the Realtime kernel entries listed on /etc/grub.conf.
    2. Restart the system to set up the reserved memory space. You can then turn on the kdump init script and start the kdump service:
      # chkconfig kdump on
      
      # service kdump status
      Kdump is not operational
      
      # service kdump start
      Starting kdump:                   [  OK  ]
      
  3. Enabling kdump with system-config-kdump

    1. Select the Kernel Crash Dumps system tool from the SystemAdministration menu, or use the following command at the shell prompt:
      # system-config-kdump
      
    2. The Kernel Dump Configuration window displays. On the toolbar, click the button labeled Enable. The MRG Realtime kernel supports the crashkernel=auto parameter which automatically calculates the amount of memory necessary to accommodate the kdump kernel.
      By design, on Red Hat Enterprise Linux 6 systems with less than 4GB of RAM, the crashkernel=auto does not reserve any memory for the kdump kernel. In this case, it is necessary to manually set the amount of memory desired. You can do so by entering your required value in the New kdump memory field on the Basic Settings tab:
      Kdump System Tool - Basic Settings

      Note

      An alternative way of allocating memory for the kdump kernel is by manually setting the crashkernel=<value> parameter on /etc/grub.conf.
    3. Click the Target Settings tab, and specify the target location for your dump file. It can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol.
      Kdump System Tool - Target Settings
      To save your settings, click the Apply button on the toolbar.
    4. Reboot the system to ensure that kdump is properly started. If you want to check that the kdump is working correctly, you can simulate a panic using sysrq:
      # echo "c" > /proc/sysrq-trigger
      
      This will cause the kernel to panic and the system will boot into the kdump kernel. Once your system has been brought back up with the boot kernel, you can check the log file at the location you specified.

Note

Some hardware needs to be reset during the configuration of the kdump kernel. If you have any problems getting the kdump kernel to work, edit the /etc/sysconfig/kdump file and add reset_devices=1 to the KDUMP_COMMANDLINE_APPEND variable.

Important

On IBM LS21 machines, the following warning message can occur when attempting to boot the kdump kernel:
irq 9: nobody cared (try booting with the "irqpoll" option) handlers:
[<ffffffff811660a0>] (acpi_irq+0x0/0x1b)
turning off IO-APIC fast mode.
Some systems will recover from this error and continue booting, while some will freeze after displaying the message. This is a known issue. If you see this error, add the line acpi=noirq as a boot parameter to the kdump kernel. Only add this line if this error occurs as it can cause boot problems on machines not affected by this issue.
Related Manual Pages
For more information, or for further reading, the following man pages are related to the information given in this section.
  • kexec(8)
  • /etc/kdump.conf