Chapter 9. Recording Application Interactions

The executable code of applications interacts with the code of the operating system and shared libraries. Recording an activity log of these interactions can provide enough insight into the application’s behavior without debugging the actual application code. Alternatively, analyzing an application’s interactions can help pinpoint the conditions in which a bug manifests.

9.1. Tools useful for recording application interactions

Red Hat Enterprise Linux offers multiple tools for analyzing an application’s interactions.

strace

The strace tool primarily enables logging of system calls (kernel functions) used by an application.

  • The strace tool can provide a detailed output about calls, because strace interprets parameters and results with knowledge of the underlying kernel code. Numbers are turned into the respective constant names, bitwise combined flags expanded to flag list, pointers to character arrays dereferenced to provide the actual string, and more. Support for more recent kernel features may be lacking.
  • You can filter the traced calls to reduce the amount of captured data.
  • The use of strace does not require any particular setup except for setting up the log filter.
  • Tracing the application code with strace results in significant slowdown of the application’s execution. As a result, strace is not suitable for many production deployments. As an alternative, consider using ltrace or SystemTap.
  • The version of strace available in Red Hat Developer Toolset can also perform system call tampering. This capability is useful for debugging.
ltrace

The ltrace tool enables logging of an application’s user space calls into shared objects (dynamic libraries).

  • The ltrace tool enables tracing calls to any library.
  • You can filter the traced calls to reduce the amount of captured data.
  • The use of ltrace does not require any particular setup except for setting up the log filter.
  • The ltrace tool is lightweight and fast, offering an alternative to strace: it is possible to trace the respective interfaces in libraries such as glibc with ltrace instead of tracing kernel functions with strace.
  • Because ltrace does not handle a known set of calls like strace, it does not attempt to explain the values passed to library functions. The ltrace output contains only raw numbers and pointers. The interpretation of ltrace output requires consulting the actual interface declarations of the libraries present in the output.
Note

In Red Hat Enterprise Linux 8.0, a known issue prevents ltrace from tracing system executable files. This limitation does not apply to executable files built by users.

SystemTap

SystemTap is an instrumentation platform for probing running processes and kernel activity on the Linux system. SystemTap uses its own scripting language for programming custom event handlers.

  • Compared to using strace and ltrace, scripting the logging means more work in the initial setup phase. However, the scripting capabilities extend SystemTap’s usefulness beyond just producing logs.
  • SystemTap works by creating and inserting a kernel module. The use of SystemTap is efficient and does not create a significant slowdown of the system or application execution on its own.
  • SystemTap comes with a set of usage examples.
GDB

The GNU Debugger (GDB) is primarily meant for debugging, not logging. However, some of its features make it useful even in the scenario where an application’s interaction is the primary activity of interest.

  • With GDB, it is possible to conveniently combine the capture of an interaction event with immediate debugging of the subsequent execution path.
  • GDB is best suited for analyzing response to infrequent or singular events, after the initial identification of problematic situation by other tools. Using GDB in any scenario with frequent events becomes inefficient or even impossible.

Additional resources

9.2. Monitoring an application’s system calls with strace

The strace tool enables monitoring the system (kernel) calls performed by an application.

Prerequisites

Procedure

  1. Identify the system calls to monitor.

  2. Start strace and attach it to the program.

    • If the program you want to monitor is not running, start strace and specify the program: 

      $ strace -fvttTyy -s 256 -e trace=call program

    • If the program is already running, find its process id (pid) and attach strace to it: 

      $ ps -C program
(...)
$ strace -fvttTyy -s 256 -e trace=call -ppid
    • Replace call with the system calls to be displayed. You can use the -e trace=call option multiple times. If left out, strace will display all system call types. See the strace(1) manual page for more information.
    • If you do not want to trace any forked processes or threads, leave out the -f option.

  3. The strace tool displays the system calls made by the application and their details.

    In most cases, an application and its libraries make a large number of calls and strace output appears immediately, if no filter for system calls is set.

  4. The strace tool exits when the program exits.

    To terminate the monitoring before the traced program exits, press Ctrl+C.

    • If strace started the program, the program terminates together with strace.
    • If you attached strace to an already running program, the program terminates together with strace.

  5. Analyze the list of system calls done by the application.

    • Problems with resource access or availability are present in the log as calls returning errors.
    • Values passed to the system calls and patterns of call sequences provide insight into the causes of the application’s behaviour.
    • If the application crashes, the important information is probably at the end of log.
    • The output contains a lot of unnecessary information. However, you can construct a more precise filter for the system calls of interest and repeat the procedure.
Note

It is advantageous to both see the output and save it to a file. Use the tee command to achieve this: 

$ strace ... |& tee your_log_file.log

Additional resources

9.3. Monitoring application’s library function calls with ltrace

The ltrace tool enables monitoring an application’s calls to functions available in libraries (shared objects).

Note

In Red Hat Enterprise Linux 8.0, a known issue prevents ltrace from tracing system executable files. This limitation does not apply to executable files built by users.

Prerequisites

Procedure

  1. Identify the libraries and functions of interest, if possible.

  2. Start ltrace and attach it to the program.

    • If the program you want to monitor is not running, start ltrace and specify program: 

      $ ltrace -f -l library -e function program

    • If the program is already running, find its process id (pid) and attach ltrace to it: 

      $ ps -C program
(...)
$ ltrace -f -l library -e function program -ppid

    • Use the -e, -f and -l options to filter the output:

      • Supply the function names to be displayed as function. The -e function option can be used multiple times. If left out, ltrace displays calls to all functions.
      • Instead of specifying functions, you can specify whole libraries with the -l library option. This option behaves similarly to the -e function option.
      • If you do not want to trace any forked processes or threads, leave out the -f option.

      See the ltrace(1)_ manual page for more information.

  3. ltrace displays the library calls made by the application.

    In most cases, an application makes a large number of calls and ltrace output displays immediately, if no filter is set.

  4. ltrace exits when the program exits.

    To terminate the monitoring before the traced program exits, press ctrl+C.

    • If ltrace started the program, the program terminates together with ltrace.
    • If you attached ltrace to an already running program, the program terminates together with ltrace.

  5. Analyze the list of library calls done by the application.

    • If the application crashes, the important information is probably at the end of log.
    • The output contains a lot of unnecessary information. However, you can construct a more precise filter and repeat the procedure.
Note

It is advantageous to both see the output and save it to a file. Use the tee command to achieve this: 

$ ltrace ... |& tee your_log_file.log

Additional resources

  • The ltrace(1) manual page: 

    $ man ltrace
  • Red Hat Developer Toolset User Guide — Chapter ltrace

9.4. Monitoring application’s system calls with SystemTap

The SystemTap tool enables registering custom event handlers for kernel events. In comparison with the strace tool, it is harder to use but more efficient and enables more complicated processing logic. A SystemTap script called strace.stp is installed together with SystemTap and provides an approximation of strace functionality using SystemTap.

Prerequisites

Procedure

  1. Find the process ID (pid) of the process you want to monitor: 

    $ ps -aux

  2. Run SystemTap with the strace.stp script: 

    # stap /usr/share/systemtap/examples/process/strace.stp -x pid

    The value of pid is the process id.

    The script is compiled to a kernel module, which is then loaded. This introduces a slight delay between entering the command and getting the output.

  3. When the process performs a system call, the call name and its parameters are printed to the terminal.
  4. The script exits when the process terminates, or when you press Ctrl+C.

9.5. Using GDB to intercept application system calls

GNU Debugger (GDB) lets you stop an execution in various situations that arise during program execution. To stop the execution when the program performs a system call, use a GDB catchpoint.

Prerequisites

Procedure

  1. Set the catchpoint: 

    (gdb) catch syscall syscall-name

    The command catch syscall sets a special type of breakpoint that halts execution when the program performs a system call.

    The syscall-name option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out the syscall-name option causes GDB to stop on any system call.

  2. Start execution of the program.

    • If the program has not started execution, start it: 

      (gdb) r

    • If the program execution is halted, resume it: 

      (gdb) c
  3. GDB halts execution after the program performs any specified system call.

Additional resources

9.6. Using GDB to intercept handling of signals by applications

GNU Debugger (GDB) lets you stop the execution in various situations that arise during program execution. To stop the execution when the program receives a signal from the operating system, use a GDB catchpoint.

Prerequisites

Procedure

  1. Set the catchpoint: 

    (gdb) catch signal signal-type

    The command catch signal sets a special type of a breakpoint that halts execution when a signal is received by the program. The signal-type option specifies the type of the signal. Use the special value 'all' to catch all signals.

  2. Let the program run.

    • If the program has not started execution, start it: 

      (gdb) r

    • If the program execution is halted, resume it: 

      (gdb) c
  3. GDB halts execution after the program receives any specified signal.

Additional resources