Chapter 9. Recording Application Interactions
The executable code of applications interacts with the code of the operating system and shared libraries. Recording an activity log of these interactions can provide enough insight into the application’s behavior without debugging the actual application code. Alternatively, analyzing an application’s interactions can help pinpoint the conditions in which a bug manifests.
9.1. Tools useful for recording application interactions
Red Hat Enterprise Linux offers multiple tools for analyzing an application’s interactions.
- strace
The
strace
tool primarily enables logging of system calls (kernel functions) used by an application.-
The
strace
tool can provide a detailed output about calls, becausestrace
interprets parameters and results with knowledge of the underlying kernel code. Numbers are turned into the respective constant names, bitwise combined flags expanded to flag list, pointers to character arrays dereferenced to provide the actual string, and more. Support for more recent kernel features may be lacking. - You can filter the traced calls to reduce the amount of captured data.
-
The use of
strace
does not require any particular setup except for setting up the log filter. -
Tracing the application code with
strace
results in significant slowdown of the application’s execution. As a result,strace
is not suitable for many production deployments. As an alternative, consider usingltrace
or SystemTap. -
The version of
strace
available in Red Hat Developer Toolset can also perform system call tampering. This capability is useful for debugging.
-
The
- ltrace
The
ltrace
tool enables logging of an application’s user space calls into shared objects (dynamic libraries).-
The
ltrace
tool enables tracing calls to any library. - You can filter the traced calls to reduce the amount of captured data.
-
The use of
ltrace
does not require any particular setup except for setting up the log filter. -
The
ltrace
tool is lightweight and fast, offering an alternative tostrace
: it is possible to trace the respective interfaces in libraries such asglibc
withltrace
instead of tracing kernel functions withstrace
. -
Because
ltrace
does not handle a known set of calls likestrace
, it does not attempt to explain the values passed to library functions. Theltrace
output contains only raw numbers and pointers. The interpretation ofltrace
output requires consulting the actual interface declarations of the libraries present in the output.
NoteIn Red Hat Enterprise Linux 8.0, a known issue prevents
ltrace
from tracing system executable files. This limitation does not apply to executable files built by users.-
The
- SystemTap
SystemTap is an instrumentation platform for probing running processes and kernel activity on the Linux system. SystemTap uses its own scripting language for programming custom event handlers.
-
Compared to using
strace
andltrace
, scripting the logging means more work in the initial setup phase. However, the scripting capabilities extend SystemTap’s usefulness beyond just producing logs. - SystemTap works by creating and inserting a kernel module. The use of SystemTap is efficient and does not create a significant slowdown of the system or application execution on its own.
- SystemTap comes with a set of usage examples.
-
Compared to using
- GDB
The GNU Debugger (GDB) is primarily meant for debugging, not logging. However, some of its features make it useful even in the scenario where an application’s interaction is the primary activity of interest.
- With GDB, it is possible to conveniently combine the capture of an interaction event with immediate debugging of the subsequent execution path.
- GDB is best suited for analyzing response to infrequent or singular events, after the initial identification of problematic situation by other tools. Using GDB in any scenario with frequent events becomes inefficient or even impossible.
Additional resources
9.2. Monitoring an application’s system calls with strace
The strace
tool enables monitoring the system (kernel) calls performed by an application.
Prerequisites
Procedure
- Identify the system calls to monitor.
Start
strace
and attach it to the program.If the program you want to monitor is not running, start
strace
and specify the program:$ strace -fvttTyy -s 256 -e trace=call program
If the program is already running, find its process id (pid) and attach
strace
to it:$ ps -C program (...) $ strace -fvttTyy -s 256 -e trace=call -ppid
-
Replace call with the system calls to be displayed. You can use the
-e trace=call
option multiple times. If left out,strace
will display all system call types. See the strace(1) manual page for more information. -
If you do not want to trace any forked processes or threads, leave out the
-f
option.
The
strace
tool displays the system calls made by the application and their details.In most cases, an application and its libraries make a large number of calls and
strace
output appears immediately, if no filter for system calls is set.The
strace
tool exits when the program exits.To terminate the monitoring before the traced program exits, press
.-
If
strace
started the program, the program terminates together withstrace
. -
If you attached
strace
to an already running program, the program terminates together withstrace
.
-
If
Analyze the list of system calls done by the application.
- Problems with resource access or availability are present in the log as calls returning errors.
- Values passed to the system calls and patterns of call sequences provide insight into the causes of the application’s behaviour.
- If the application crashes, the important information is probably at the end of log.
- The output contains a lot of unnecessary information. However, you can construct a more precise filter for the system calls of interest and repeat the procedure.
It is advantageous to both see the output and save it to a file. Use the tee
command to achieve this:
$ strace ... |& tee your_log_file.log
Additional resources
The strace(1) manual page:
$ man strace
- How do I use strace to trace system calls made by a command? — Knowledgebase article
- Red Hat Developer Toolset User Guide — Chapter strace
9.3. Monitoring application’s library function calls with ltrace
The ltrace
tool enables monitoring an application’s calls to functions available in libraries (shared objects).
In Red Hat Enterprise Linux 8.0, a known issue prevents ltrace
from tracing system executable files. This limitation does not apply to executable files built by users.
Prerequisites
Procedure
- Identify the libraries and functions of interest, if possible.
Start
ltrace
and attach it to the program.If the program you want to monitor is not running, start
ltrace
and specify program:$ ltrace -f -l library -e function program
If the program is already running, find its process id (pid) and attach
ltrace
to it:$ ps -C program (...) $ ltrace -f -l library -e function program -ppid
Use the
-e
,-f
and-l
options to filter the output:-
Supply the function names to be displayed as function. The
-e function
option can be used multiple times. If left out,ltrace
displays calls to all functions. -
Instead of specifying functions, you can specify whole libraries with the
-l library
option. This option behaves similarly to the-e function
option. -
If you do not want to trace any forked processes or threads, leave out the
-f
option.
See the ltrace(1)_ manual page for more information.
-
Supply the function names to be displayed as function. The
ltrace
displays the library calls made by the application.In most cases, an application makes a large number of calls and
ltrace
output displays immediately, if no filter is set.ltrace
exits when the program exits.To terminate the monitoring before the traced program exits, press
.-
If
ltrace
started the program, the program terminates together withltrace
. -
If you attached
ltrace
to an already running program, the program terminates together withltrace
.
-
If
Analyze the list of library calls done by the application.
- If the application crashes, the important information is probably at the end of log.
- The output contains a lot of unnecessary information. However, you can construct a more precise filter and repeat the procedure.
It is advantageous to both see the output and save it to a file. Use the tee
command to achieve this:
$ ltrace ... |& tee your_log_file.log
Additional resources
The ltrace(1) manual page:
$ man ltrace
- Red Hat Developer Toolset User Guide — Chapter ltrace
9.4. Monitoring application’s system calls with SystemTap
The SystemTap tool enables registering custom event handlers for kernel events. In comparison with the strace
tool, it is harder to use but more efficient and enables more complicated processing logic. A SystemTap script called strace.stp
is installed together with SystemTap and provides an approximation of strace
functionality using SystemTap.
Procedure
Find the process ID (pid) of the process you want to monitor:
$ ps -aux
Run SystemTap with the
strace.stp
script:# stap /usr/share/systemtap/examples/process/strace.stp -x pid
The value of pid is the process id.
The script is compiled to a kernel module, which is then loaded. This introduces a slight delay between entering the command and getting the output.
- When the process performs a system call, the call name and its parameters are printed to the terminal.
-
The script exits when the process terminates, or when you press
Ctrl+C
.
9.5. Using GDB to intercept application system calls
GNU Debugger (GDB) lets you stop an execution in various situations that arise during program execution. To stop the execution when the program performs a system call, use a GDB catchpoint.
Prerequisites
Procedure
Set the catchpoint:
(gdb) catch syscall syscall-name
The command
catch syscall
sets a special type of breakpoint that halts execution when the program performs a system call.The
syscall-name
option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out thesyscall-name
option causes GDB to stop on any system call.Start execution of the program.
If the program has not started execution, start it:
(gdb) r
If the program execution is halted, resume it:
(gdb) c
- GDB halts execution after the program performs any specified system call.
Additional resources
9.6. Using GDB to intercept handling of signals by applications
GNU Debugger (GDB) lets you stop the execution in various situations that arise during program execution. To stop the execution when the program receives a signal from the operating system, use a GDB catchpoint.
Prerequisites
Procedure
Set the catchpoint:
(gdb) catch signal signal-type
The command
catch signal
sets a special type of a breakpoint that halts execution when a signal is received by the program. Thesignal-type
option specifies the type of the signal. Use the special value'all'
to catch all signals.Let the program run.
If the program has not started execution, start it:
(gdb) r
If the program execution is halted, resume it:
(gdb) c
- GDB halts execution after the program receives any specified signal.
Additional resources