Why kdump service fails to build initramfs if "path" is not set in /etc/kdump.conf file.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.1
  • kexec-tools

Issue

  • kdump service fails to build initramfs if "path" is not set in /etc/kdump.conf file.
  • Kdump fails with below call traces

    # systemctl status kdump.service
    kdump.service - Crash recovery kernel arming
    Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled)
    Active: failed (Result: exit-code) since Tue 2015-07-28 04:00:54 PDT; 6s ago
    Process: 30341 ExecStop=/usr/bin/kdumpctl stop (code=exited, status=0/SUCCESS)
    Process: 30350 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
    Main PID: 30350 (code=exited, status=1/FAILURE)
    
    Jul 28 04:00:53 hostname kdumpctl[30350]: Detected change(s) in the following file(s):
    Jul 28 04:00:53 hostname kdumpctl[30350]: /etc/kdump.conf
    Jul 28 04:00:53 hostname kdumpctl[30350]: Rebuilding /boot/initramfs-3.10.0-229.el7.x86_64kdump.img
    Jul 28 04:00:54 hostname kdumpctl[30350]: Dump path /var/crash//var/crash/ does not exist.
    Jul 28 04:00:54 hostname kdumpctl[30350]: mkdumprd: failed to make kdump initrd
    Jul 28 04:00:54 hostname kdumpctl[30350]: Starting kdump: [FAILED]
    Jul 28 04:00:54 hostname systemd[1]: kdump.service: main process exited, code=exited, status=1/FAILURE
    Jul 28 04:00:54 hostname systemd[1]: Failed to start Crash recovery kernel arming.
    Jul 28 04:00:54 hostname systemd[1]: Unit kdump.service entered failed state.
    

Resolution

  • This is an expected behavior.

    • If kdump service is configured to dump the vmcore on NFS share then it is necessary to set the "path" variable in /etc.kdump.conf file else "mkdumprd" command will fail to generate a initramfs file for kdump kernel.
    • This behavior is introduced in /usr/sbin/mkdumprd bash script provided by "kexec-tools" package shipped with RHEL-7.1.
    • In older version of kexec-tools package if /var/crash directory is not present on NFS share then kdump service will automatically create it without any error message.
  • This is documented in /usr/share/doc/kexec-tools-2.0.7/kexec-kdump-howto.txt

    path /var/crash/ (NFS mounted on /var)
    
    Say foo.com:/export/tmp is mounted on /var. In this case dump target is
    nfs server and path will be adjusted to "/crash" and dump will be saved 
    to foo.com:/export/tmp/crash/ directory.
    

    Resolution:

    • Set "path" to "/var/crash" in /etc/kdump.conf file if NFS share is mounted on "/var".
      OR
    • Set "path" to "/" in /etc/kdump.conf file if NFS share is mounted on "/var/crash".

Root Cause

  • In older version of kexec-tools package if /var/crash directory is not present on NFS
    share then kdump service will automatically create it without any error message.
    RHEL-7.0 [ /usr/sbin/mkdumprd ]

    ext[234]|xfs|btrfs|minix|nfs)             <<<---
        if ! findmnt $config_val >/dev/null; then
            perror_exit "Dump target $config_val is probably not mounted."
        fi
    
        if [ "$config_opt" = "nfs" ]; then
            add_dracut_module "nfs"
        fi
        add_mount "$config_val"
        mkdir_save_path_fs $config_val    <<<---
        check_size fs $config_val
        ;;
    
  • In newer version of kexec-tools package We need to set the "path" in /etc/kdump.conf file
    to dump the vmcore on NFS share else kdump service will fail with an error message.
    RHEL-7.1 [ /usr/sbin/mkdumprd ]

     ext[234]|xfs|btrfs|minix|nfs)                                       <<<---
        if ! findmnt $config_val >/dev/null; then
            perror_exit "Dump target $config_val is probably not mounted."
        fi
    
        if [ "$config_opt" = "nfs" ]; then
            add_dracut_module "nfs"
        fi
        add_mount "$config_val"
        check_save_path_fs $(make_absolute_save_path $config_val)   <<<---
        check_size fs $config_val
        ;;
         check_save_path_fs()
       {
             local _path=$1
             if [ ! -d $_path ]; then
             perror_exit "Dump path $_path does not exist."             <<<---
             fi
       }
    
  • This is documented in /usr/share/doc/kexec-tools-2.0.7/kexec-kdump-howto.txt

      path /var/crash/ (NFS mounted on /var)
      ----------------------------------------------
     Say foo.com:/export/tmp is mounted on /var. In this case dump target is
     nfs server and path will be adjusted to "/crash" and dump will be saved 
     to foo.com:/export/tmp/crash/ directory.
    

Diagnostic Steps

  • Check kdump Configuration.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.