Chapter 20. Debugging a Running Application

This chapter will introduce the techniques for debugging an application which can be started as many times as needed, on a machine directly accessible to the developer.

20.1. Enabling Debugging with Debugging Information

To debug applications and libraries, debugging information is required. The following sections describe how to obtain this information.

20.1.1. Debugging Information

While debugging any executable code, two kinds of information allow the tools and by extension the programmer to comprehend the binary code:

  • the source code text
  • a description of how the source code text relates to the binary code

This is referred to as debugging information.

Red Hat Enterprise Linux uses the ELF format for executable binaries, shared libraries, or debuginfo files. Within these ELF files, the DWARF format is used to hold the debug information.

DWARF symbols are read by the readelf -w file command.

Caution

STABS is occasionally used with UNIX. STABS is an older, less capable format. Its use is discouraged by Red Hat. GCC and GDB support STABS production and consumption on a best effort basis only. Some other tools such as Valgrind and elfutils do not support STABS at all.

Additional Resources

20.1.2. Enabling Debugging of C and C++ Applications with GCC

Because debugging information is large, it is not included in executable files by default. To enable debugging of your C and C++ applications with it, you must explicitly instruct the compiler to create it.

Enabling Creation of Debugging Information with GCC

To enable creation of debugging information with GCC when compiling and linking code, use the -g option: 

$ gcc ... -g ...
  • Optimizations performed by the compiler and linker can result in executable code which is hard to relate to the original source code: Variables may be optimized out, loops unrolled, operations merged into the surrounding ones etc. This affects debugging negatively. For improved debuging experience, consider setting the optimization with the -Og option. However, changing the optimization level changes the executable code and may change the actual behaviour so as to remove some bugs.
  • The -fcompare-debug GCC option tests code compiled by GCC with debug information and without debug information. The test passes if the resulting two binary files are identical. This test ensures that executable code is not affected by any debugging options, which further ensures that there are no hidden bugs in the debug code. Note that using the -fcompare-debug option significantly increases compilation time. See the GCC manual page for details about this option.
Additional Resources

20.1.3. Debuginfo Packages

Debuginfo packages contain debugging information and debug source code for programs and libraries.

Prerequisites
Debuginfo Packages

For applications and libraries installed in packages from the Red Hat Enterprise Linux repositories, you can obtain the debugging information and debug source code as separate debuginfo packages available through another channel. The debuginfo packages contain .debug files, which contain DWARF debuginfo and the source files used for compiling the binary packages. Debuginfo package contents are installed to the /usr/lib/debug directory.

A debuginfo package provides debugging information valid only for a binary package with the same name, version, release and architecture:

  • Binary package: packagename-version-release.architecture.rpm
  • Debuginfo package: packagename-debuginfo-version-release.architecture.rpm

20.1.4. Getting debuginfo Packages for an Application or Library using GDB

The GNU Debugger (GDB) automatically recognizes missing debug information and resolves the package name.

Prerequisites
Procedure

  1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. 

    $ gdb -q /bin/ls
Reading symbols from /usr/bin/ls...Reading symbols from /usr/bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install coreutils-8.22-21.el7.x86_64
(gdb)

  2. Exit GDB without proceeding further: type q and Enter. 

    (gdb) q

  3. Run the command suggested by GDB to install the needed debuginfo packages: 

    # debuginfo-install coreutils-8.22-21.el7.x86_64

    Installing a debuginfo package for an application or library installs debuginfo packages for all dependencies, too.

  4. In case GDB is not able to suggest the debuginfo package, follow the procedure in Section 20.1.5, “Getting debuginfo Packages for an Application or Library Manually”.
Additional Resources

20.1.5. Getting debuginfo Packages for an Application or Library Manually

You can determine manually debuginfo packages for installation by locating the executable file and finding the package which installs it.

Note

Prefer use of GDB to determine the packages for installation. Use this manual procedure only if GDB is not able to suggest the package to install.

Prerequisites
Procedure

  1. Find the executable file of the application or library.

    1. Use the which command to find the application file. 

      $ which nautilus
/usr/bin/nautilus

    2. Use the locate command to find the library file. 

      $ locate libz | grep so
/usr/lib64/libz.so
/usr/lib64/libz.so.1
/usr/lib64/libz.so.1.2.7

      If the original reasons for debugging included error messages, pick the result where the library has the same additional numbers in its file name. If in doubt, try following the rest of the procedure with the result where the library file name includes no additional numbers.

      Note

      The locate command is provided by the mlocate package. To install it and enable its use: 

      # yum install mlocate
# updatedb

  2. Using the file path, search for a package which provides that file. 

    # yum provides /usr/lib64/libz.so.1.2.7
Loaded plugins: product-id, search-disabled-repos, subscription-manager
zlib-1.2.7-17.el7.x86_64 : The compression and decompression library
Repo        : @anaconda/7.4
Matched from:
Filename    : /usr/lib64/libz.so.1.2.7

    The output provides a list of packages in the format name-version.distribution.platform. In this step, only the package name is important, because the version shown in yum output may not be the actual installed version.

    Important

    If this step does not produce any results, it is not possible to determine which package provided the binary file and this procedure fails.

  3. Use the rpm low-level package management tool to find what package version is installed on the system. Use the package name as an argument: 

    $ rpm -q zlib
zlib-1.2.7-17.el7.x86_64

    The output provides details for the installed package in the format name-version.distribution.platform.

  4. Install the debuginfo packages using the debuginfo-install utility. In the command, use the package name and other details you determined during the previous step: 

    # debuginfo-install zlib-1.2.7-17.el7.x86_64

    Installing a debuginfo package for an application or library installs debuginfo packages for all dependencies, too.

Additional Resources

20.2. Inspecting Application Internal State with GDB

To find why an application does not work properly, control its execution and examine its internal state with a debugger. This section describes how to use the GNU Debugger (GDB) for this task.

20.2.1. GNU Debugger (GDB)

A debugger is a tool that enables control of code execution and inspection of the state of the code. This capability is used to investigate what is happening in a program and why.

Red Hat Enterprise Linux contains the GNU debugger (GDB) which offers this functionality through a command line user interface.

For a graphical frontend to GDB, install the Eclipse integrated development environment. See Using Eclipse.

GDB Capabilities

A single GDB session can debug:

  • multithreaded and forking programs
  • multiple programs at once
  • programs on remote machines or in containers with the gdbserver utility connected over a TCP/IP network connection
Debugging Requirements

To debug any executable code, GDB requires the respective debugging information:

  • For programs developed by you, you can create the debugging information while building the code.
  • For system programs installed from packages, their respective debuginfo packages must be installed.

20.2.2. Attaching GDB to a Process

In order to examine a process, GDB must be attached to the process.

Prerequisites
Starting a Program with GDB

When the program is not running as a process, start it with GDB: 

$ gdb program

Replace program with file name or path to the program.

GDB sets up to start execution of the program. You can set up breakpoints and the gdb environemnt before beginning execution of the process with the run command.

Attaching GDB to an Already Running Process

To attach GDB to a program already running as a process:

  1. Find the process id (pid) with the ps command: 

    $ ps -C program -o pid h
 pid

    Replace program with file name or path to the program.

  2. Attach GDB to this process: 

    $ gdb -p pid

    Replace pid with an actual process id number from the ps output.

Attaching an Already Running GDB to an Already Running Process

To attach an already running GDB to an already running program:

  1. Use the shell GDB command to run the ps command and find the program’s process id (pid): 

    (gdb) shell ps -C program -o pid h
 pid

    Replace program with file name or path to the program.

  2. Use the attach command to attach GDB to the program: 

    (gdb) attach pid

    Replace pid by an actual process id number from the ps output.

Note

In some cases, GDB might not be able to find the respective executable file. Use the file command to specify the path: 

(gdb) file path/to/program
Additional Resources

20.2.3. Stepping through Program Code with GDB

Once the GDB debugger is attached to a program, you can use a number of commands to control the execution of the program.

Prerequisites
GDB Commands to Step Through the Code
r (run)
Start the execution of the program. If run is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally. Users normally issue this command after setting breakpoints.
start
Start the execution of the program, and stop at the beginning of the program’s main function. If start is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally.
c (continue)

Continue the execution of the program from the current state. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
n (next)

Continue the execution of the program from the current state, until next line of code in the current source file is reached. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
s (step)
The step command also halts execution at each sequential line of code in the current source file. However, if the execution is currently stopped at a source line containing a function call, GDB stops the execution after entering the function call (rather than executing it).
until location
Continue the execution until the code location specified by the location option is reached.
fini (finish)

Resume the execution of the program and halt when execution returns from a function. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
q (quit)
Terminate the execution and exit GDB.
Additional Resources

20.2.4. Showing Program Internal Values with GDB

Displaying thevalues of a program’s internal variables is important for understanding of what the program is doing. GDB offers multiple commands that you can use to inspect the internal variables. This section describes the most useful of these commands.

Prerequisites
  • Understanding the GDB debugger
GDB Commands to Display the Internal State of a Program
p (print)

Display the value of the argument given. Usually, the argument is the name of a variable of any complexity, from a simple single value to a structure. An argument can also be an expression valid in the current language, including the use of program variables and library functions, or functions defined in the program being tested.

It is possible to extend GDB with pretty-printer Python or Guile scripts for customized display of data structures (such as classes, structs) using the print command.

bt (backtrace)

Display the chain of function calls used to reach the current execution point, or the chain of functions used up until execution was terminated. This is useful for investigating serious bugs (such as segmentation faults) with elusive causes.

Adding the full option to the backtrace command displays local variables, too.

It is possible to extend GDB with frame filter Python scripts for customized display of data displayed using the bt and info frame commands. The term frame refers to the data associated with a single function call.

info

The info command is a generic command to provide information about various items. It takes an option specifying the item to describe.

  • The info args command displays options of the function call that is the currently selected frame.
  • The info locals command displays local variables in the currently selected frame.

For a list of the possible items, run the command help info in a GDB session: 

(gdb) help info
l (list)
Show the line in the source code where the program stopped. This command is available only when the program execution is stopped. While not strictly a command to show internal state, list helps the user understand what changes to the internal state will happen in the next step of the program’s execution.
Additional Resources

20.2.5. Using GDB Breakpoints to Stop Execution at Defined Code Locations

In many cases, it is advantageous to let the program execute until a certain line of code is reached.

Prerequisites
  • Understanding GDB
Using Breakpoints in GDB

Breakpoints are markers that tell GDB to stop the execution of a program. Breakpoints are most commonly associated with source code lines: Placing a breakpoint requires specifying the source file and line number.

  • To place a breakpoint:

    • Specify the name of the source code file and the line in that file: 

      (gdb) br file:line

    • When file is not present, name of the source file at the current point of execution is used: 

      (gdb) br line

    • Alternatively, use a function name to put the breakpoint on its start: 

      (gdb) br function_name

  • A program might encounter an error after a certain number of iterations of a task. To specify an additional condition to halt execution: 

    (gdb) br file:line if condition

    Replace condition with a condition in the C or C++ language. The meaning of file and line is the same as above.

  • To inspect the status of all breakpoints and watchpoints: 

    (gdb) info br

  • To remove a breakpoint by using its number as displayed in the output of info br: 

    (gdb) delete number

  • To remove a breakpoint at a given location: 

    (gdb) clear file:line
Additional Resources

20.2.6. Using GDB Watchpoints to Stop Execution on Data Access and Changes

In many cases, it is advantageous to let the program execute until certain data changes or is accessed. This section lists the most common

Prerequisites
  • Understanding GDB
Using Watchpoints in GDB

Watchpoints are markers which tell GDB to stop the execution of program. Watchpoints are associated with data: Placing a watchpoint requires specifying an expression describing a variable, multiple variables, or a memory address.

  • To place a watchpoint for data change (write): 

    (gdb) watch expression

    Replace expression with an expression that describes what you want to watch. For variables, expression is equal to the name of the variable.

  • To place a watchpoint for data access (read): 

    (gdb) rwatch expression

  • To place a watchpoint for any data access (both read and write): 

    (gdb) awatch expression

  • To inspect the status of all watchpoints and breakpoints: 

    (gdb) info br

  • To remove a watchpoint: 

    (gdb) delete num

    Replace the num option with the number reported by the info br command.

Additional Resources

20.2.7. Debugging Forking or Threaded Programs with GDB

Some programs use forking or threads to achieve parallel code execution. Debugging multiple simultaneous execution paths requires special considerations.

Prerequisites
  • Understanding the GDB debugger
  • Understanding the concepts of process forking and threads
Debugging Forked Programs with GDB

Forking is a situation when a program (parent) creates an independent copy of itself (child). Use the following settings and commands to affect what GDB does when a fork occurs:

  • The follow-fork-mode setting controls whether GDB follows the parent or the child after the fork.

    set follow-fork-mode parent
    After a fork, debug the parent process. This is the default.
    set follow-fork-mode child
    After a fork, debug the child process.
    show follow-fork-mode
    Display the current setting of follow-fork-mode.

  • The set detach-on-fork setting controls whether the GDB keeps control of the other (not followed) process or leaves it to run.

    set detach-on-fork on
    The process which is not followed (depending on the value of follow-fork-mode) is detached and runs independently. This is the default.
    set detach-on-fork off
    GDB keeps control of both processes. The process which is followed (depending on the value of follow-fork-mode) is debugged as usual, while the other is suspended.
    show detach-on-fork
    Display the current setting of detach-on-fork.
Debugging Threaded Programs with GDB

GDB has the ability to debug individual threads, and to manipulate and examine them independently. To make GDB stop only the thread that is examined, use the commands set non-stop on and set target-async on. You can add these commands to the .gdbinit file. After that functionality is turned on, GDB is ready to conduct thread debugging.

GDB uses a concept of current thread. By default, commands apply to the current thread only.

info threads
Display a list of threads with their id and gid numbers, indicating the current thread.
thread id
Set the thread with the specified id as the current thread.
thread apply ids command
Apply the command command to all threads listed by ids. The ids option is a space-separated list of thread ids. A special value all applies the command to all threads.
break location thread id if condition
Set a breakpoint at a certain location with a certain condition only for the thread number id.
watch expression thread id
Set a watchpoint defined by expression only for the thread number id.
command&
Execute command command and return immediately to the gdb prompt (gdb), continuing any code execution in the background.
interrupt
Halt execution in the background.
Additional Resources

20.3. Recording Application Interactions

The executable code of applications interacts with the code of the operating system and shared libraries. Recording an activity log of these interactions can provide enough insight into the application’s behavior without debugging the actual application code. Alternatively, analyzing an application’s interactions can help pinpoint the conditions in which a bug manifests.

20.3.1. Tools Useful for Recording Application Interactions

Red Hat Enterprise Linux offers multiple tools for analyzing an application’s interactions should be analyzed.

strace

The strace tool primarily enables logging of system calls (kernel functions) used by an application.

  • The strace output is detailed and explains the calls well, because strace interprets parameters and results with knowledge of the underlying kernel code. Numbers are turned into the respective constant names, bitwise combined flags expanded to flag list, pointers to character arrays dereferenced to provide the actual string, and more. Support for more recent kernel features may be lacking.
  • You can filter the traced calls to reduce the amount of captured data.
  • The use of [command]`strace does not require any particular setup except for setting up the log filter.
  • Tracing the application code with strace results in significant slowdown of the application’s execution. As a result, strace is not suitable for many production deployments. As an alternative, consider using ltrace or SystemTap.
  • The version of strace available in Red Hat Developer Toolset can also perform system call tampering. This capability is useful for debugging.
ltrace

The ltrace tool enables logging of an application’s user space calls into shared objects (dynamic libraries).

  • ltrace enables tracing calls to any library.
  • You can filter the traced calls to reduce the amount of captured data.
  • The use of ltrace does not require any particular setup except for setting up the log filter.
  • ltrace is lightweight and fast, offering an alternative to strace: it is possible to trace the respective interfaces in libraries such as glibc with ltrace instead of tracing kernel functions with strace.
  • Because ltrace does not handle a known set of calls like strace, it does not attempt to explain the values passed to library functions. The ltrace output contains only raw numbers and pointers. The interpretation of ltrace output requires consulting the actual interface declarations of the libraries present in the output.
SystemTap

SystemTap is an instrumentation platform for probing running processes and kernel activity on the Linux system. SystemTap uses its own scripting language for programming custom event handlers.

  • Compared to using strace and ltrace, scripting the logging means more work in the initial setup phase. However, the scripting capabilities extend SystemTap’s usefulness beyond just producing logs.
  • SystemTap works by creating and inserting a kernel module. The use of SystemTap is efficient and does not create a significant slowdown of the system or application execution on its own.
  • SystemTap comes with a set of usage examples.
GDB

The GNU Debugger is primarily meant for debugging, not logging. However, some of its features make it useful even in the scenario where an application’s interaction is the primary activity of interest.

  • With GDB, it is possible to conveniently combine the capture of an interaction event with immediate debugging of the subsequent execution path.
  • GDB is best suited for analyzing response to infrequent or singular events, after the initial identification of problematic situation by other tools. Using GDB in any scenario with frequent events becomes inefficient or even impossible.
Additional Resources

20.3.2. Monitoring an Application’s System Calls with strace

The strace tool enables monitoring the system (kernel) calls performed by an application.

Prerequisites
Steps
  1. Identify the system calls you wish to monitor.

  2. If the program you want to monitor is not running, start strace and specify the program: 

    $ strace -fvttTyy -s 256 -e trace=call program

    Replace call with the system calls to be displayed. You can use the -e trace=call option multiple times. If left out, strace will display all system call types. See the strace(1) manual page for more information.

    If the program is already running, find its process id (pid) and attach strace to it: 

    $ ps -C program
(...)
$ strace -fvttTyy -s 256 -e trace=call -ppid

    If you do not wish to trace any forked processes or threads, leave out the -f option.

  3. strace displays the system calls made by the application and their details.

    In most cases, an application and its libraries make a large number of calls and strace output appears immediately, if no filter for system calls is set.

  4. strace exits when the program exits.

    To terminate the monitoring before the traced program exits, press ctrl+C.

    • If strace started the program, the program terminates together with strace.
    • If you attached strace to an already running program, the program terminates together with strace.

  5. Analyze the list of system calls done by the application.

    • Problems with resource access or availability are present in the log as calls returning errors.
    • Values passed to the system calls and patterns of call sequences provide insight into the causes of the application’s behaviour.
    • If the application crashes, the important information is probably at the end of log.
    • The output contains a lot of unnecessary information. However, you can construct a more precise filter and repeat the procedure.
Note

It is advantageous to both see the output and save it to a file. Use the tee command to achieve this: 

$ strace ... |& tee your_log_file.log
Additional Resources

20.3.3. Monitoring Application’s Library Function Calls with ltrace

The ltrace tool enables monitoring of the calls done by an application to functions available in libraries (shared objects).

Prerequisites
Steps
  1. Identify the libraries and functions of interest, if possible.

  2. If the program you want to monitor is not running, start ltrace and specify program: 

    $ ltrace -f -l library -e function program

    Use the options -e and -l to filter the output:

    • Supply the function names to be displayed as function. The -e function option can be used multiple times. If left out, ltrace will display calls to all functions.
    • Instead of specifying functions, you can specify whole libraries with the -l library option. This option behaves similarly to the -e function option.

    See the ltrace(1)_ manual page for more information.

    If the program is already running, find its process id (pid) and attach ltrace to it: 

    $ ps -C program
(...)
$ ltrace ... -ppid

    If you do not wish to trace any forked processes or threads, leave out the -f option.

  3. ltrace displays the library calls made by the application.

    In most cases, an application will make a large number of calls and ltrace output appears immediately, if no filter is set.

  4. ltrace exits when the program exits.

    To terminate the monitoring before the traced program exits, press ctrl+C.

    • If ltrace started the program, the program terminates together with ltrace.
    • If you attached ltrace to an already running program, the program terminates together with ltrace.

  5. Analyze the list of library calls done by the application.

    • If the application crashes, the important information is probably at the end of log.
    • The output contains a lot of unnecessary information. However, you can construct a more precise filter and repeat the procedure.
Note

It is advantageous to both see the output and save it to a file. Use the tee command to achieve this: 

$ ltrace ... |& tee your_log_file.log
Additional Resources
  • The strace(1) manual page.
  • Red Hat Developer Toolset User Guide — Chapter ltrace

20.3.4. Monitoring Application’s System Calls with SystemTap

The SystemTap tool enables registering custom event handlers for kernel events. In comparison with strace, it is harder to use but more efficient and enables more complicated processing logic.

Prerequisites
Steps

  1. Create a file my_script.stp with the contents: 

    probe begin
{
  printf("waiting for syscalls of process %d \n", target())
}

probe syscall.*
{
  if (pid() == target())
    printf("%s(%s)\n", name, argstr)
}

probe process.end
{
  if (pid() == target())
    exit()
}

  2. Find the process ID (pid) of the process you wish to monitor: 

    $ ps -aux

  3. Run SystemTap with the script: 

    # stap my_script.stp -x pid

    The value of pid is the process id.

    The script is compiled to a kernel module which is then loaded. This introduces a slight delay between entering the command and getting the output.

  4. When the process performs a system call, the call name and its parameters are printed to the terminal.
  5. The script exits when the process terminates, or when you press Ctrl+C.
Additional Resources

20.3.5. Using GDB to Intercept Application System Calls

GDB enables stopping the execution in various kinds of situations arising during the execution of a program. To stop the execution when the program performs a system call, use a GDB catchpoint.

Prerequisites

Stopping Program Execution on System Calls with GDB

  1. Set the catchpoint: 

    (gdb) catch syscall syscall-name

    The command catch syscall sets a special type of breakpoint that halts execution when a system call is performed by the program.

    The syscall-name option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out the syscall-name option causes GDB to stop on any system call.

  2. If the program has not started execution, start it: 

    (gdb) r

    If the program execution is only halted, resume it: 

    (gdb) c
  3. GDB halts execution after any specified system call is performed by the program.
Additional Resources

20.3.6. Using GDB to Intercept Handling of Signals by Applications

GDB enables stopping the execution in various kinds of situations arising during the execution of a program. To stop the execution when the program receives a signal from the operating system, use a GDB catchpoint.

Prerequisites
Stopping Program Execution on Receiving a Signal with GDB

  1. Set the catchpoint: 

    (gdb) catch signal signal-type

    The command catch signal sets a special type of a breakpoint that halts execution when a signal is received by the program. The signal-type option specifies the type of the signal. Use the special value 'all' to catch all signals.

  2. If the program has not started execution, start it: 

    (gdb) r

    If the program execution is only halted, resume it: 

    (gdb) c
  3. GDB halts execution after the program receives any specified signal.
Additional Resources