Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Chapter 21. Debugging a Crashed Application

Sometimes, it is not possible to debug an application directly. In these situations, you can collect information about the application at the moment of its termination and analyze it afterwards.

21.1. Core Dumps

This section describes what a core dump is and how to use it.

Prerequisites

  • Understanding of debugging information

Description

A core dump is a copy of a part of the application’s memory at the moment the application stopped working, stored in the ELF format. It contains all the application’s internal variables and stack, which enables inspection of the application’s final state. When augmented with the respective executable file and debugging information, it is possible to analyze a core dump file with a debugger in a way similar to analyzing a running program.

The Linux operating system kernel can record core dumps automatically, if this functionality is enabled. Alternatively, you can send a signal to any running application to generate a core dump regardless of its actual state.

Warning

Some limits might affect the ability to generate a core dump.

21.2. Recording Application Crashes with Core Dumps

To record application crashes, set up core dump saving and add information about the system.

Procedure

  1. Enable core dumps. Edit the file /etc/systemd/system.conf and change the line containing DefaultLimitCORE to the following: 

    DefaultLimitCORE=infinity

  2. Reboot the system: 

    # shutdown -r now

  3. Remove the limits for core dump sizes: 

    # ulimit -c unlimited

    To reverse this change, run the command with the value 0 instead of unlimited.

  4. When an application crashes, a core dump is generated. The default location for core dumps is the application’s working directory at the time of the crash.

  5. Create an SOS report to provide additional information about the system: 

    # sosreport

    This creates a tar archive containing information about your system, such as copies of configuration files.

  6. Transfer the core dump and the SOS report to the computer where the debugging will take place. Transfer the executable file, too, if it is known.

    Important

    If the executable file is not known, subsequent analysis of the core file will identify it.

  7. Optional: Remove the core dump and SOS report after transferring them to free up disk space.

Additional Resources

21.3. Inspecting Application Crash States with Core Dumps

Prerequisites

  • You have a core dump file and SOS report
  • GDB and elfutils are installed on the system

Procedure

  1. To identify the executable file where the crash occurred, run the eu-unstrip command with the core dump file: 

    $ eu-unstrip -n --core=./core.9814
0x400000+0x207000 2818b2009547f780a5639c904cded443e564973e@0x400284 /usr/bin/sleep /usr/lib/debug/bin/sleep.debug [exe]
0x7fff26fff000+0x1000 1e2a683b7d877576970e4275d41a6aaec280795e@0x7fff26fff340 . - linux-vdso.so.1
0x35e7e00000+0x3b6000 374add1ead31ccb449779bc7ee7877de3377e5ad@0x35e7e00280 /usr/lib64/libc-2.14.90.so /usr/lib/debug/lib64/libc-2.14.90.so.debug libc.so.6
0x35e7a00000+0x224000 3ed9e61c2b7e707ce244816335776afa2ad0307d@0x35e7a001d8 /usr/lib64/ld-2.14.90.so /usr/lib/debug/lib64/ld-2.14.90.so.debug ld-linux-x86-64.so.2

    The output contains details for each module on one line, separated by spaces. The information is listed in this order:

    1. The memory address where the module was mapped
    2. The build-id of the module and where in the memory it was found
    3. The module’s executable file name, displayed as - when unknown, or as . when the module has not been loaded from a file
    4. The source of debugging information, displayed as a file name when available, as . when contained in the executable file itself, or as - when not present at all
    5. The shared library name (soname), or [exe] for the main module

    In this example, the important details are the file name /usr/bin/sleep and the build-id 2818b2009547f780a5639c904cded443e564973e on the line containing the text [exe]. With this information, you can identify the executable file required for analyzing the core dump.

  2. Get the executable file that crashed.

    • If possible, copy it from the system where the crash occurred. Use the file name extracted from the core file.

    • Alternatively, use an identical executable file on your system. Each executable file built on Red Hat Enterprise Linux contains a note with a unique build-id value. Determine the build-id of the relevant locally available executable files: 

      $ eu-readelf -n executable_file

      Use this information to match the executable file on the remote system with your local copy. The build-id of the local file and build-id listed in the core dump must match.

    • Finally, if the application is installed from an RPM package, you can get the executable file from the package. Use the sosreport output to find the exact version of the package required.
  3. Get the shared libraries used by the executable file. Use the same steps as for the executable file.
  4. If the application is distributed as a package, load the executable file in GDB to display hints for missing debuginfo packages. For more details, see Section 20.1.4, “Getting debuginfo Packages for an Application or Library using GDB”.

  5. To examine the core file in detail, load the executable file and core dump file with GDB: 

    $ gdb -e executable_file -c core_file

    Further messages about missing files and debugging information help you to identify what is missing for the debugging session. Return to the previous step if needed.

    If the debugging information is available as a file instead of a package, load this file in GDB with the symbol-file command: 

    (gdb) symbol-file program.debug

    Replace program.debug with the actual file name.

    Note

    It might not be necessary to install the debugging information for all executable files contained in the core dump. Most of these executable files are libraries used by the application code. These libraries might not directly contribute to the problem you are analyzing, and you do not need to include debugging information for them.

  6. Use the GDB commands to inspect the state of the application at the moment it crashed. See Section 20.2, “Inspecting the Application’s Internal State with GDB”.

    Note

    When analyzing a core file, GDB is not attached to a running process. Commands for controlling execution have no effect.

Additional Resources

21.4. Dumping Process Memory with gcore

The workflow of core dump debugging enables the analysis of the offline state of the program. In some cases it is advantageous to use this workflow with a program that is still running, such as when it is hard to access the environment with the process. You can use the gcore command to dump memory of any process while it is still running.

Prerequisites

Procedure

To dump a process memory using gcore:

  1. Find out the process id (pid). Use tools such as ps, pgrep, and top: 

    $ ps -C some-program

  2. Dump the memory of this process: 

    $ gcore -o filename pid

    This creates a file filename and dumps the process memory in it. While the memory is being dumped, the execution of the process is halted.

  3. After the core dump is finished, the process resumes normal execution.

  4. Create an SOS report to provide additional information about the system: 

    # sosreport

    This creates a tar archive containing information about your system, such as copies of configuration files.

  5. Transfer the program’s executable file, core dump, and the SOS report to the computer where the debugging will take place.
  6. Optional: Remove the core dump and SOS report after transferring them to reclaim disk space.

Additional resources

21.5. Dumping Protected Process Memory with GDB

You can mark the memory of processes as not to be dumped. This can save resources and ensure additional security if the process memory contains sensitive data. Both kernel core dumps (kdump) and manual core dumps (gcore, GDB) do not dump memory marked this way.

In some cases, it is necessary to dump the whole contents of the process memory regardless of these protections. This procedure shows how to do this using the GDB debugger.

Prerequisites

Procedure

  1. Set GDB to ignore the settings in the /proc/PID/coredump_filter file: 

    (gdb) set use-coredump-filter off

  2. Set GDB to ignore the memory page flag VM_DONTDUMP: 

    (gdb) set dump-excluded-mappings on

  3. Dump the memory: 

    (gdb) gcore core-file

    Replace core-file with the name of the file of which you want to dump the memory.

Additional Resources