Chapter 8. Inspecting Application Internal State with GDB

To find why an application does not work properly, control its execution and examine its internal state with a debugger. This section describes how to use the GNU Debugger (GDB) for this task.

8.1. GNU debugger (GDB)

Red Hat Enterprise Linux contains the GNU debugger (GDB) which lets you investigate what is happening inside a program through a command line user interface.

For a graphical frontend to GDB, install the Eclipse integrated development environment. See Using Eclipse.

GDB capabilities

A single GDB session can debug the following types of programs:

  • Multithreaded and forking programs
  • Multiple programs at once
  • Programs on remote machines or in containers with the gdbserver utility connected over a TCP/IP network connection

Debugging requirements

To debug any executable code, GDB requires debugging information for that particular code:

  • For programs developed by you, you can create the debugging information while building the code.
  • For system programs installed from packages, you must install their debuginfo packages.

8.2. Attaching GDB to a process

In order to examine a process, GDB must be attached to the process.

Starting a program with GDB

When the program is not running as a process, start it with GDB:

$ gdb program

Replace program with file name or path to the program.

GDB sets up to start execution of the program. You can set up breakpoints and the gdb environemnt before beginning execution of the process with the run command.

Attaching GDB to an already running process

To attach GDB to a program already running as a process:

  1. Find the process id (pid) with the ps command:

    $ ps -C program -o pid h
     pid

    Replace program with file name or path to the program.

  2. Attach GDB to this process:

    $ gdb -p pid

    Replace pid with an actual process id number from the ps output.

Attaching an already running GDB to an already running process

To attach an already running GDB to an already running program:

  1. Use the shell GDB command to run the ps command and find the program’s process id (pid):

    (gdb) shell ps -C program -o pid h
     pid

    Replace program with file name or path to the program.

  2. Use the attach command to attach GDB to the program:

    (gdb) attach pid

    Replace pid by an actual process id number from the ps output.

Note

In some cases, GDB might not be able to find the respective executable file. Use the file command to specify the path:

(gdb) file path/to/program

Additional resources

8.3. Stepping through program code with GDB

Once the GDB debugger is attached to a program, you can use a number of commands to control the execution of the program.

Prerequisites

GDB commands to step through the code

r (run)
Start the execution of the program. If run is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally. Users normally issue this command after setting breakpoints.
start
Start the execution of the program, and stop at the beginning of the program’s main function. If start is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally.
c (continue)

Continue the execution of the program from the current state. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
n (next)

Continue the execution of the program from the current state, until next line of code in the current source file is reached. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
s (step)
The step command also halts execution at each sequential line of code in the current source file. However, if the execution is currently stopped at a source line containing a function call, GDB stops the execution after entering the function call (rather than executing it).
until location
Continue the execution until the code location specified by the location option is reached.
fini (finish)

Resume the execution of the program and halt when execution returns from a function. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached
  • A specified condition is satisfied
  • A signal is received by the program
  • An error occurs
  • The program terminates
q (quit)
Terminate the execution and exit GDB.

8.4. Showing program internal values with GDB

Displaying thevalues of a program’s internal variables is important for understanding of what the program is doing. GDB offers multiple commands that you can use to inspect the internal variables. This section describes the most useful of these commands:

p (print)

Display the value of the argument given. Usually, the argument is the name of a variable of any complexity, from a simple single value to a structure. An argument can also be an expression valid in the current language, including the use of program variables and library functions, or functions defined in the program being tested.

It is possible to extend GDB with pretty-printer Python or Guile scripts for customized display of data structures (such as classes, structs) using the print command.

bt (backtrace)

Display the chain of function calls used to reach the current execution point, or the chain of functions used up until execution was terminated. This is useful for investigating serious bugs (such as segmentation faults) with elusive causes.

Adding the full option to the backtrace command displays local variables, too.

It is possible to extend GDB with frame filter Python scripts for customized display of data displayed using the bt and info frame commands. The term frame refers to the data associated with a single function call.

info

The info command is a generic command to provide information about various items. It takes an option specifying the item to describe.

  • The info args command displays options of the function call that is the currently selected frame.
  • The info locals command displays local variables in the currently selected frame.

For a list of the possible items, run the command help info in a GDB session:

(gdb) help info
l (list)
Show the line in the source code where the program stopped. This command is available only when the program execution is stopped. While not strictly a command to show internal state, list helps the user understand what changes to the internal state will happen in the next step of the program’s execution.

Additional resources

8.5. Using GDB breakpoints to stop execution at defined code locations

Investigations of code are often concerned about only a small portion of the code. Breakpoints are markers that tell GDB to stop the execution of a program at a certain place in the code. Breakpoints are most commonly associated with source code lines. In that case, placing a breakpoint requires specifying the source file and line number.

  • To place a breakpoint:

    • Specify the name of the source code file and the line in that file:

      (gdb) br file:line
    • When file is not present, name of the source file at the current point of execution is used:

      (gdb) br line
    • Alternatively, use a function name to put the breakpoint on its start:

      (gdb) br function_name
  • A program might encounter an error after a certain number of iterations of a task. To specify an additional condition to halt execution:

    (gdb) br file:line if condition

    Replace condition with a condition in the C or C++ language. The meaning of file and line is the same as above.

  • To inspect the status of all breakpoints and watchpoints:

    (gdb) info br
  • To remove a breakpoint by using its number as displayed in the output of info br:

    (gdb) delete number
  • To remove a breakpoint at a given location:

    (gdb) clear file:line

Additional resources

8.6. Using GDB watchpoints to stop execution on data access and changes

In many cases, it is advantageous to let the program execute until certain data changes or is accessed. This section lists the most common

Prerequisites

  • Understanding GDB

Using watchpoints in GDB

Watchpoints are markers which tell GDB to stop the execution of program. Watchpoints are associated with data: Placing a watchpoint requires specifying an expression describing a variable, multiple variables, or a memory address.

  • To place a watchpoint for data change (write):

    (gdb) watch expression

    Replace expression with an expression that describes what you want to watch. For variables, expression is equal to the name of the variable.

  • To place a watchpoint for data access (read):

    (gdb) rwatch expression
  • To place a watchpoint for any data access (both read and write):

    (gdb) awatch expression
  • To inspect the status of all watchpoints and breakpoints:

    (gdb) info br
  • To remove a watchpoint:

    (gdb) delete num

    Replace the num option with the number reported by the info br command.

Additional resources

8.7. Debugging forking or threaded programs with GDB

Some programs use forking or threads to achieve parallel code execution. Debugging multiple simultaneous execution paths requires special considerations.

Prerequisites

  • You must understand the concepts of process forking and threads.

Debugging forked programs with GDB

Forking is a situation when a program (parent) creates an independent copy of itself (child). Use the following settings and commands to affect what GDB does when a fork occurs:

  • The follow-fork-mode setting controls whether GDB follows the parent or the child after the fork.

    set follow-fork-mode parent
    After a fork, debug the parent process. This is the default.
    set follow-fork-mode child
    After a fork, debug the child process.
    show follow-fork-mode
    Display the current setting of follow-fork-mode.
  • The set detach-on-fork setting controls whether the GDB keeps control of the other (not followed) process or leaves it to run.

    set detach-on-fork on
    The process which is not followed (depending on the value of follow-fork-mode) is detached and runs independently. This is the default.
    set detach-on-fork off
    GDB keeps control of both processes. The process which is followed (depending on the value of follow-fork-mode) is debugged as usual, while the other is suspended.
    show detach-on-fork
    Display the current setting of detach-on-fork.

Debugging Threaded Programs with GDB

GDB has the ability to debug individual threads, and to manipulate and examine them independently. To make GDB stop only the thread that is examined, use the commands set non-stop on and set target-async on. You can add these commands to the .gdbinit file. After that functionality is turned on, GDB is ready to conduct thread debugging.

GDB uses a concept of current thread. By default, commands apply to the current thread only.

info threads
Display a list of threads with their id and gid numbers, indicating the current thread.
thread id
Set the thread with the specified id as the current thread.
thread apply ids command
Apply the command command to all threads listed by ids. The ids option is a space-separated list of thread ids. A special value all applies the command to all threads.
break location thread id if condition
Set a breakpoint at a certain location with a certain condition only for the thread number id.
watch expression thread id
Set a watchpoint defined by expression only for the thread number id.
command&
Execute command command and return immediately to the gdb prompt (gdb), continuing any code execution in the background.
interrupt
Halt execution in the background.

Additional resources