kdump fails to start with the error "Could not find a free area of memory of 3b1000 bytes... locate_hole failed"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6.1

Issue

  • When the crashkernel memory reservation goes above 896M, kdump fails to start with the following message:
    # /sbin/kexec -p '--command-line=ro root=/dev/mapper/vg_system-root rd_LVM_LV=vg_system/root rd_LVM_LV=vg_system/swap rd_NO_LUKS rd_NO_MD 
    rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fr console=ttyS1,9600n8 irqpoll maxcpus=1 reset_devices 
    cgroup_disable=memory ' --initrd=/boot/initrd-2.6.32-131.0.15.el6.x86_64kdump.img /boot/vmlinuz-2.6.32-131.0.15.el6.x86_64
    Could not find a free area of memory of 3b1000 bytes...
    locate_hole failed

Resolution

  • Kdump will never need more than 896MB for it's operation. In most of the cases 256MB is enough, and even systems with 1TB of memory will work fine with a reservation of 512MB. 
  • This requirement comes even further down with the changes proposed in RHEL 6.2 (one of the major change is the switch from mxacpus=1 to nr_cpus=1)

Root Cause

  • 64-bit kernel starts executing in real- or ia32-mode. When crashed kernel wants to jump into kdump kernel, it switches to 32-bit mode via purgatory code and then jumps to new kernel (any kernel image should be in first 4 gigs). If crashkernel reserved area can't be fully fitted in 4G, kexec fails to load kdump kernel (I/O regions, like GART, also reduce available space). The purgatory code has hand-written assembly code with relocations of type R_X86_64_32S (32 bit signed). This cuts down suitable space for purgatory code to 2Gb. Again kexec for x86_64 shares a lot of code with i386, and it has hardcoded address limits for kernel and initrd images at 896 Mb. In practice, suitable space for kdump kernel is 1-896 mb.

Diagnostic Steps

  • Set the kdump init script into DEBUG mode:
# head -1 /etc/rc.d/init.d/kdump
#! /bin/sh -x
  • Kdump service failing with below error:
+ /usr/bin/logger -p info -t kdump 'kexec: failed to load kdump kernel'
  • Try manually loading the kexec kernel:
    # /sbin/kexec -p '--command-line=ro root=/dev/mapper/vg_system-root rd_LVM_LV=vg_system/root rd_LVM_LV=vg_system/swap rd_NO_LUKS rd_NO_MD 
    rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=fr console=ttyS1,9600n8 irqpoll maxcpus=1 reset_devices 
    cgroup_disable=memory ' --initrd=/boot/initrd-2.6.32-131.0.15.el6.x86_64kdump.img /boot/vmlinuz-2.6.32-131.0.15.el6.x86_64
    Could not find a free area of memory of 3b1000 bytes...
    locate_hole failed

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments