Chapter 12. SystemTap

SystemTap is a tracing and probing tool that allows users to monitor the activities of the entire system without needing to instrument, recompile, install, and reboot. It is programmable with a custom scripting language, which gives it expressiveness (to trace, filter, and analyze) and reach (to look into the running kernel and applications).

SystemTap can monitor various types of events, such as function calls within the kernel or applications, timers, tracepoints, performance counters, and so on. Some included example scripts produce output similar to netstat, ps, top, and iostat, others include pretty-printed function callgraph traces or tools for working around security bugs.

Red Hat Developer Toolset is distributed with SystemTap 4.1. This version is more recent than the version included in the previous release of Red Hat Developer Toolset and provides numerous bug fixes and enhancements.

Table 12.1. Tools Distributed with SystemTap for Red Hat Developer Toolset

NameDescription

stap

Translates probing instructions into C code, builds a kernel module, and loads it into a running Linux kernel.

stapdyn

The Dyninst backend for SystemTap.

staprun

Loads, unloads, attaches to, and detaches from kernel modules built with the stap utility.

stapsh

Serves as a remote shell for SystemTap.

stap-prep

Determines and—if possible—downloads the kernel information packages that are required to run SystemTap.

stap-merge

Merges per-CPU files. This script is automatically executed when the stap utility is executed with the -b command line option.

stap-report

Gathers important information about the system for the purpose of reporting a bug in SystemTap.

stap-server

A compile server, which listens for requests from stap clients.

12.1. Installing SystemTap

In Red Hat Developer Toolset, SystemTap is provided by the devtoolset-9-systemtap package and is automatically installed with devtoolset-9-perftools as described in Section 1.5, “Installing Red Hat Developer Toolset”.

In order to place instrumentation into the Linux kernel, SystemTap may also require installation of additional packages with debugging information. To determine which packages to install, run the stap-prep utility as follows: 

$ scl enable devtoolset-9 'stap-prep'

Note that if you execute this command as the root user, the utility automatically offers the packages for installation. For more information on how to install these packages on your system, see the Red Hat Enterprise Linux 7 SystemTap Beginners Guide.

12.2. Using SystemTap

To execute any of the tools that are part of SystemTap: 

$ scl enable devtoolset-9 'tool option...'

See Table 12.1, “Tools Distributed with SystemTap for Red Hat Developer Toolset” for a list of tools that are distributed with SystemTap. For example, to run the stap tool to build an instrumentation module: 

$ scl enable devtoolset-9 'stap option... argument...'

Note that you can execute any command using the scl utility, causing it to be run with the Red Hat Developer Toolset binaries used in preference to the Red Hat Enterprise Linux system equivalent. This allows you to run a shell session with Red Hat Developer Toolset SystemTap as default: 

$ scl enable devtoolset-9 'bash'
Note

To verify the version of SystemTap you are using at any point: 

$ which stap

Red Hat Developer Toolset’s stap executable path will begin with /opt. Alternatively, you can use the following command to confirm that the version number matches that for Red Hat Developer Toolset SystemTap: 

$ stap -V

12.3. Additional Resources

For more information about SystemTap and its features, see the resources listed below.

Installed Documentation

  • stap(1) — The manual page for the stap command provides detailed information on its usage, as well as references to other related manual pages. To display the manual page for the version included in Red Hat Developer Toolset: 

    $ scl enable devtoolset-9 'man stap'

  • staprun(8) — The manual page for the staprun command provides detailed information on its usage. To display the manual page for the version included in Red Hat Developer Toolset: 

    $ scl enable devtoolset-9 'man staprun'

Online Documentation

See Also

  • Chapter 1, Red Hat Developer Toolset — An overview of Red Hat Developer Toolset and more information on how to install it on your system.
  • Chapter 13, Valgrind — Instructions on using the Valgrind tool to profile applications and detect memory errors and memory management problems, such as the use of uninitialized memory, improper allocation and freeing of memory, and the use of improper arguments in system calls.
  • Chapter 14, OProfile — Instructions on using the OProfile tool to determine which sections of code consume the greatest amount of CPU time and why.
  • Chapter 15, Dyninst — Instructions on using the Dyninst library to instrument a user-space executable.