What is SystemTap and how to use it?
Prerequisites
This knowledge applies to the following Linux versions.
- RHEL 5.3 Server and later
- RHEL 6.0 Server and later
- RHEL 7.0 Server and later
A summary about SystemTap is available at the following solution. This article is to complement it.
1. Required packages
SystemTap uses two RHEL servers in general. One is to build a kernel module from your SystemTap scripts. The other is to be analyzed with that module. You can build modules on the RHEL server which you want to analyze, but in production systems, you may want to use a different RHEL server for building modules because you need additional development packages to build modules.
The following package and its dependent ones should be installed in the RHEL server to be analyzed.
- systemtap-runtime(*1)
The following is an example of confirming if the required package is installed. In addition, please check the kernel version as you will need it later.
| $ rpm -q systemtap-runtime systemtap-runtime-2.3-4.el6_5.x86_64 $ uname -r 2.6.32-431.11.2.el6.x86_64 |
The following packages and their dependent ones should be installed in the RHEL server to build kernel modules.
An example of confirmation result is shown below.
| $ rpm -q systemtap systemtap-runtime gcc kernel-devel kernel-debuginfo systemtap-2.3-4.el6_5.x86_64 systemtap-runtime-2.3-4.el6_5.x86_64 gcc-4.4.7-4.el6.x86_64 kernel-devel-2.6.32-431.11.2.el6.x86_64 kernel-debuginfo-2.6.32-431.11.2.el6.x86_64 |
(*1)SystemTap's major version can be different within RHEL's minor releases because Systemtap is under rapid development to provide more powerful debugging capability. In older SystemTap and its related packages, some syntax and/or options may not be supported, or its behavior might be different. We recommend using the latest packages supported by your RHEL major release.
| Version of RHEL release | Version of systemtap related package included |
|---|---|
| 5.0 | 0.5.12-1.el5 |
| 5.1 | 0.5.14-1.el5 |
| 5.2 | 0.6.2-1.el5 |
| 5.3 | 0.7.2-2.el5 |
| 5.4 | 0.9.7-5.el5 |
| 5.5 | 1.1-3.el5 |
| 5.6 | 1.3-4.el5 |
| 5.7 | 1.3-8.el5 |
| 5.8 | 1.6-6.el5 |
| 5.9 | 1.8-6.el5 |
| 5.10 | 1.8-6.el5 |
| 5.11 | 1.8-6.el5 |
| 6.0 | 1.2-9.el6 |
| 6.1 | 1.4-6.el6 |
| 6.2 | 1.6-4.el6 |
| 6.3 | 1.7-5.el6 |
| 6.4 | 1.8-7.el6 |
| 6.5 | 2.3-3.el6 | 6.6 | 2.5-5.el6 |
| 7.0 | 2.4-14.el7 |
| 7.1 | 2.6-8.el7 |
(*2) The links for downloading RHEL's packages are listed below. Please note that you need to download and install versions which matches the architecture and version of the kernel used in the environment to obtain information.
| RHEL 7 / x86_64 | |
|---|---|
| kernel-devel kernel-debuginfo kernel-debuginfo-common-x86_64 |
|
| RHEL 6 / x86_64 | RHEL 6 / i386 |
| kernel-devel kernel-debuginfo kernel-debuginfo-common-x86_64 |
kernel-devel kernel-debuginfo kernel-debuginfo-common-i686 |
| RHEL 5 / Other than Xen / x86_64 | RHEL 5 / Other than Xen / i386 |
| kernel-devel kernel-debuginfo kernel-debuginfo-common |
kernel-devel kernel-debuginfo kernel-debuginfo-common |
| RHEL 5 / Xen / x86_64 | RHEL 5 / Xen / i386 |
| kernel-xen-devel kernel-xen-debuginfo kernel-debuginfo-common |
kernel-xen-devel kernel-xen-debuginfo kernel-debuginfo-common |
2. Writing SystemTap scripts and building modules.
The content of SystemTap script you need to write depends on the information you want to obtain from the system, and its kernel version. As an explanation purpose, we assume to use the following example.
| example.stp - Notify when a thread exits. |
|---|
| probe begin { printf("Probe started.\n"); } probe kernel.function("do_exit") { printf("%s PID=%u TID=%u COMM=%s exited.\n", ctime(gettimeofday_s()), pid(), tid(), execname()); } probe end { printf("Probe ended.\n"); } |
To compile the script, run stap command as below.
| $ stap -p4 -r $kernelversion -m stap_example example.stp |
-p4 is an option to proceed the stages up to compiling a kernel module. You can compile a kernel module with an unprivileged user.
-r is an option to specify the kernel version used in the RHEL server to be analyzed.
-m is an option to specify the name of kernel module to be generated.
While you can specify arbitrary name as long as the name is valid as kernel module's name, it is recommended to use names prefixed by stap_ so that everyone can understand that it is a kernel module used by SystemTap.
Upon successful compilation, a kernel module with the filename specified with -m option followed by .ko is generated.
The following is an example for compiling for 2.6.32-431.11.2.el6.x86_64 kernel. Depending on the content of script, some other options may be required.
| $ stap -p4 -r 2.6.32-431.11.2.el6.x86_64 -m stap_example example.stp stap_example.ko |
If the RHEL server to compile kernel modules is different from the RHEL server you want to analyze, please copy the generated kernel module (stap_example.ko for the example above) to the RHEL server to be analyzed.
Tips
If the kernel version used by the environment running stap command and that of the environment to obtain information are identical, you can omit -r $kernelversion option when you run stap command.
Please use systemtap-1.1-3.el5 (or later) and the -a option followed by the architecture name (the output of uname -i command) if the architecture of the environment to compile kernel modules and that of the environment to obtain information differs.
3. Running the SystemTap scripts
Run staprun command as root user on the RHEL server to be analyzed.
| # staprun $path_to_kernel_module |
An example of execution result is shown below.
| [root@localhost ~]# staprun stap_example.ko Probe started. Wed Apr 23 03:50:01 2014 PID=2486 TID=2486 COMM=sadc exited. Wed Apr 23 03:50:01 2014 PID=2485 TID=2485 COMM=crond exited. Wed Apr 23 03:50:06 2014 PID=1875 TID=1875 COMM=sleep exited. Wed Apr 23 03:50:06 2014 PID=2488 TID=2488 COMM=awk exited. Wed Apr 23 03:50:06 2014 PID=2487 TID=2487 COMM=ksmtuned exited. Wed Apr 23 03:50:06 2014 PID=2491 TID=2491 COMM=pgrep exited. |
You can terminate staprun process using Ctrl-C.
Please note that staprun process may automatically terminate due to SystemTap's safety mechanisms and/or too many events to process have occurred. If the staprun process terminates before collecting information or capturing events you want, you need to modify your SystemTap script and/or change compile options.
Tips
If you compile kernel modules directly on the RHEL server to be analyzed, you can run staprun command by omitting the -p4 option so that it automatically starts the analysis.
| # stap example.stp |
You can use -o option followed by output filename if you want to save stdout of the command.
| # staprun -o $output_file $path_to_kernel_module |
Additional information
SystemTap's sample scripts are available at the following location.
These sample scripts will help you understand what you can do using SystemTap.
Please use them at your own risk with accepting implications that you might need to install newer versions of systemtap packages for compilation which are not included in your RHEL major release and/or you might not be able to obtain expected results even if you successfully compiled them.
Comments