Chapter 11. SystemTap
SystemTap is a tracing and probing tool that allows users to monitor the activities of the entire system without needing to instrument, recompile, install, and reboot. It is programmable with a custom scripting language, which gives it expressiveness (to trace, filter, and analyze) and reach (to look into the running kernel and applications).
SystemTap can monitor various types of events, such as function calls within the kernel or applications, timers, tracepoints, performance counters, and so on. Some included example scripts produce output similar to
iostat, others include pretty-printed function callgraph traces or tools for working around security bugs.
Red Hat Developer Toolset is distributed with SystemTap 3.1. This version is more recent than the version included in the previous release of Red Hat Developer Toolset and provides numerous bug fixes and enhancements.
Table 11.1. Tools Distributed with SystemTap for Red Hat Developer Toolset
| || |
Translates probing instructions into C code, builds a kernel module, and loads it into a running Linux kernel.
| || |
The Dyninst backend for SystemTap.
| || |
Loads, unloads, attaches to, and detaches from kernel modules built with the
| || |
Serves as a remote shell for SystemTap.
| || |
Determines and—if possible—downloads the kernel information packages that are required to run SystemTap.
| || |
Merges per-CPU files. This script is automatically executed when the
| || |
Gathers important information about the system for the purpose of reporting a bug in SystemTap.
| || |
A compile server, which listens for requests from
11.1. Installing SystemTap
In Red Hat Developer Toolset,
SystemTap is provided by the devtoolset-7-systemtap package and is automatically installed with devtoolset-7-perftools as described in Section 1.5, “Installing Red Hat Developer Toolset”.
The Red Hat Developer Toolset version of SystemTap is available for both Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7, but some new features are only offered by the Red Hat Developer Toolset version of SystemTap for Red Hat Enterprise Linux 7.
In order to place instrumentation into the Linux kernel, SystemTap may also require installation of additional packages with debugging information. To determine which packages to install, run the
stap-prep utility as follows:
scl enable devtoolset-7 'stap-prep'
Note that if you execute this command as the
root user, the utility automatically offers the packages for installation. For more information on how to install these packages on your system, see the Red Hat Enterprise Linux 6 SystemTap Beginners Guide or the Red Hat Enterprise Linux 7 SystemTap Beginners Guide.
11.2. Using SystemTap
To execute any of the tools that are part of SystemTap:
scl enable devtoolset-7 'tool option...'
See Table 11.1, “Tools Distributed with SystemTap for Red Hat Developer Toolset” for a list of tools that are distributed with SystemTap. For example, to run the
stap tool to build an instrumentation module:
scl enable devtoolset-7 'stap option... argument...'
Note that you can execute any command using the
scl utility, causing it to be run with the Red Hat Developer Toolset binaries used in preference to the Red Hat Enterprise Linux system equivalent. This allows you to run a shell session with Red Hat Developer Toolset SystemTap as default:
scl enable devtoolset-7 'bash'
To verify the version of SystemTap you are using at any point:
Red Hat Developer Toolset’s
stap executable path will begin with
/opt. Alternatively, you can use the following command to confirm that the version number matches that for Red Hat Developer Toolset SystemTap:
11.3. Additional Resources
A detailed description of SystemTap and its features is beyond the scope of this book. For more information, see the resources listed below.
stap(1) — The manual page for the
stapcommand provides detailed information on its usage, as well as references to other related manual pages. To display the manual page for the version included in Red Hat Developer Toolset:
scl enable devtoolset-7 'man stap'
staprun(8) — The manual page for the
stapruncommand provides detailed information on its usage. To display the manual page for the version included in Red Hat Developer Toolset:
scl enable devtoolset-7 'man staprun'
SystemTap Tapset Reference Manual — HTML documentation on the most common tapset definitions is located at
- Red Hat Enterprise Linux 6 SystemTap Beginners Guide and Red Hat Enterprise Linux 7 SystemTap Beginners Guide — The SystemTap Beginners Guides for Red Hat Enterprise Linux 6 and 7 provide an introduction to SystemTap and its usage.
- Red Hat Enterprise Linux 6 SystemTap Tapset Reference and Red Hat Enterprise Linux 7 SystemTap Tapset Reference — The SystemTap Tapset Reference for Red Hat Enterprise Linux 6 and 7 provides further details about SystemTap.
- The SystemTap Documentation — The official SystemTap documentation provides further documentation on SystemTap, as well as numerous examples of SystemTap scripts.
- Section A.2.9, “Changes in SystemTap” — A comprehensive list of features and improvements over the version of SystemTap distributed in the previous release of Red Hat Developer Toolset.
- Chapter 1, Red Hat Developer Toolset — An overview of Red Hat Developer Toolset and more information on how to install it on your system.
- Chapter 12, Valgrind — Instructions on using the Valgrind tool to profile applications and detect memory errors and memory management problems, such as the use of uninitialized memory, improper allocation and freeing of memory, and the use of improper arguments in system calls.
- Chapter 13, OProfile — Instructions on using the OProfile tool to determine which sections of code consume the greatest amount of CPU time and why.
- Chapter 14, Dyninst — Instructions on using the Dyninst library to instrument a user-space executable.