What is SystemTap and how to use it?

Updated -

Prerequisites

This knowledge applies to the following Linux versions.

  • RHEL 5.3 Server and later
  • RHEL 6.0 Server and later
  • RHEL 7.0 Server and later

A summary about SystemTap is available at the following solution. This article is to complement it.

1. Required packages

SystemTap uses two RHEL servers in general. One is to build a kernel module from your SystemTap scripts. The other is to be analyzed with that module. You can build modules on the RHEL server which you want to analyze, but in production systems, you may want to use a different RHEL server for building modules because you need additional development packages to build modules.

The following package and its dependent ones should be installed in the RHEL server to be analyzed.

  • systemtap-runtime(*1)

The following is an example of confirming if the required package is installed. In addition, please check the kernel version as you will need it later.

$ rpm -q systemtap-runtime
systemtap-runtime-2.3-4.el6_5.x86_64
$ uname -r
2.6.32-431.11.2.el6.x86_64

The following packages and their dependent ones should be installed in the RHEL server to build kernel modules.

  • systemtap, systemtap-runtime(*1)
  • gcc
  • kernel-devel, kernel-debuginfo, kernel-debuginfo-common (*2)

An example of confirmation result is shown below.

$ rpm -q systemtap systemtap-runtime gcc kernel-devel kernel-debuginfo
systemtap-2.3-4.el6_5.x86_64
systemtap-runtime-2.3-4.el6_5.x86_64
gcc-4.4.7-4.el6.x86_64
kernel-devel-2.6.32-431.11.2.el6.x86_64
kernel-debuginfo-2.6.32-431.11.2.el6.x86_64

 

(*1)SystemTap's major version can be different within RHEL's minor releases because Systemtap is under rapid development to provide more powerful debugging capability. In older SystemTap and its related packages, some syntax and/or options may not be supported, or its behavior might be different. We recommend using the latest packages supported by your RHEL major release.

Version of RHEL release Version of systemtap related package included
5.0 0.5.12-1.el5
5.1 0.5.14-1.el5
5.2 0.6.2-1.el5
5.3 0.7.2-2.el5
5.4 0.9.7-5.el5
5.5 1.1-3.el5
5.6 1.3-4.el5
5.7 1.3-8.el5
5.8 1.6-6.el5
5.9 1.8-6.el5
5.10 1.8-6.el5
5.11 1.8-6.el5
6.0 1.2-9.el6
6.1 1.4-6.el6
6.2 1.6-4.el6
6.3 1.7-5.el6
6.4 1.8-7.el6
6.5 2.3-3.el6
6.6 2.5-5.el6
7.0 2.4-14.el7
7.1 2.6-8.el7


(*2) The links for downloading RHEL's packages are listed below. Please note that you need to download and install versions which matches the architecture and version of the kernel used in the environment to obtain information.

RHEL 7 / x86_64
kernel-devel
kernel-debuginfo
kernel-debuginfo-common-x86_64
RHEL 6 / x86_64 RHEL 6 / i386
kernel-devel
kernel-debuginfo
kernel-debuginfo-common-x86_64
kernel-devel
kernel-debuginfo
kernel-debuginfo-common-i686
RHEL 5 / Other than Xen / x86_64 RHEL 5 / Other than Xen / i386
kernel-devel
kernel-debuginfo
kernel-debuginfo-common
kernel-devel
kernel-debuginfo
kernel-debuginfo-common
RHEL 5 / Xen / x86_64 RHEL 5 / Xen / i386
kernel-xen-devel
kernel-xen-debuginfo
kernel-debuginfo-common
kernel-xen-devel
kernel-xen-debuginfo
kernel-debuginfo-common


2. Writing SystemTap scripts and building modules.

The content of SystemTap script you need to write depends on the information you want to obtain from the system, and its kernel version. As an explanation purpose, we assume to use the following example.

example.stp - Notify when a thread exits.
probe begin { printf("Probe started.\n"); }
probe kernel.function("do_exit") {
  printf("%s PID=%u TID=%u COMM=%s exited.\n",
          ctime(gettimeofday_s()), pid(), tid(), execname());
}
probe end { printf("Probe ended.\n"); }

To compile the script, run stap command as below.

$ stap -p4 -r $kenrelversion -m stap_example example.stp

-p4 is an option to proceed the stages up to compiling a kernel module. You can compile a kernel module with an unprivileged user.

-r is an option to specify the kernel version used in the RHEL server to be analyzed.

-m is an option to specify the name of kernel module to be generated.

While you can specify arbitrary name as long as the name is valid as kernel module's name, it is recommended to use names prefixed by stap_ so that everyone can understand that it is a kernel module used by SystemTap.

Upon successful compilation, a kernel module with the filename specified with -m option followed by .ko is generated.

The following is an example for compiling for 2.6.32-431.11.2.el6.x86_64 kernel. Depending on the content of script, some other options may be required.

$ stap -p4 -r 2.6.32-431.11.2.el6.x86_64 -m stap_example example.stp
stap_example.ko

If the RHEL server to compile kernel modules is different from the RHEL server you want to analyze, please copy the generated kernel module (stap_example.ko for the example above) to the RHEL server to be analyzed.

Tips

If the kernel version used by the environment running stap command and that of the environment to obtain information are identical, you can omit -r $kenrelversion option when you run stap command.

Please use systemtap-1.1-3.el5 (or later) and the -a option followed by the architecture name (the output of uname -i command) if the architecture of the environment to compile kernel modules and that of the environment to obtain information differs.


3. Running the SystemTap scripts

Run staprun command as root user on the RHEL server to be analyzed.

# staprun $path_to_kernel_module

An example of execution result is shown below.

[root@localhost ~]# staprun stap_example.ko
Probe started.
Wed Apr 23 03:50:01 2014 PID=2486 TID=2486 COMM=sadc exited.
Wed Apr 23 03:50:01 2014 PID=2485 TID=2485 COMM=crond exited.
Wed Apr 23 03:50:06 2014 PID=1875 TID=1875 COMM=sleep exited.
Wed Apr 23 03:50:06 2014 PID=2488 TID=2488 COMM=awk exited.
Wed Apr 23 03:50:06 2014 PID=2487 TID=2487 COMM=ksmtuned exited.
Wed Apr 23 03:50:06 2014 PID=2491 TID=2491 COMM=pgrep exited.

You can terminate staprun process using Ctrl-C.

Please note that staprun process may automatically terminate due to SystemTap's safety mechanisms and/or too many events to process have occurred. If the staprun process terminates before collecting information or capturing events you want, you need to modify your SystemTap script and/or change compile options.

Tips

If you compile kernel modules directly on the RHEL server to be analyzed, you can run staprun command by omitting the -p4 option so that it automatically starts the analysis.

# stap example.stp

You can use -o option followed by output filename if you want to save stdout of the command.

# staprun -o $output_file $path_to_kernel_module


Additional information

SystemTap's sample scripts are available at the following location.

These sample scripts will help you understand what you can do using SystemTap.

Please use them at your own risk with accepting implications that you might need to install newer versions of systemtap packages for compilation which are not included in your RHEL major release and/or you might not be able to obtain expected results even if you successfully compiled them.