Why is kdump generating incomplete vmcore files?

Solution Verified - Updated -

Environment

Red Hat Enterprise Linux (RHEL)

Issue

Under which conditions does kdump generate incomplete vmcore files?

Resolution

An incomplete vmcore is generated when dumping operation is interrupted. The common causes of incomplete dumps are:

  • Exhaustion of free space on kdump target. Even if page exclusion and core compression are enabled, the only sure way to guarantee a successful vmcore is to have free space on disk at least equal to physical RAM.
  • Cluster or hardware mechanisms rebooting the server while dumping is in progress. If a cluster or hardware-based watchdog timer mechanisms are present, make sure that the delay before reboot will be triggered is adequate to complete the dumping process.
  • Local or remote kdump target becomes inaccessible.
  • kdump kernel crashes while dumping is in progress.
  • kdump kernel runs out of memory.
  • When incomplete vmcores are generated, the console logs collected while triggering a kernel panic frequently help in finding why incomplete vmcores are generated.

Root Cause

When Kdump is configured properly, a secondary kernel, called capture kernel, is loaded inside the reserved memory area. When the system crashes, in other words when the production kernel is no longer working, kexec loads the capture kernel from the context of the production kernel.
After that, makedumpfile start to coping the memory dump in /proc/vmcore to the default target in disk. If the copy process fails, an incomplete vmcore will be generated. It may happen due to low disk space or a fencing event, for instance.

Diagnostic Steps

  • Check if there is enough disk space.
  • Make sure that fencing events won't interfere in makedumpfile copy process.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments