Red Hat Enterprise Linux 7

SystemTap Beginners Guide

Introduction to SystemTap

William Cohen

Red Hat Software Engineering

Don Domingo

Red Hat Customer Content Services

Robert Kratky

Red Hat Customer Content Services

Jacquelynn East

Red Hat Customer Content Services


This guide provides basic instructions on how to use SystemTap to monitor different subsystems of Red Hat Enterprise Linux 7 in detail. The SystemTap Beginners Guide is recommended for users who have taken the RHCSA exam or have a similar level of expertise in Red Hat Enterprise Linux 7.
1. Introduction
1.1. Documentation Goals
1.2. SystemTap Capabilities
2. Using SystemTap
2.1. Installation and Setup
2.1.1. Installing SystemTap
2.1.2. Installing Required Kernel Information Packages
2.1.3. Initial Testing
2.2. Generating Instrumentation for Other Computers
2.3. Running SystemTap Scripts
2.3.1. SystemTap Flight Recorder Mode
3. Understanding How SystemTap Works
3.1. Architecture
3.2. SystemTap Scripts
3.2.1. Event
3.2.2. Systemtap Handler/Body
3.3. Basic SystemTap Handler Constructs
3.3.1. Variables
3.3.2. Conditional Statements
3.3.3. Command-Line Arguments
3.4. Associative Arrays
3.5. Array Operations in SystemTap
3.5.1. Assigning an Associated Value
3.5.2. Reading Values From Arrays
3.5.3. Incrementing Associated Values
3.5.4. Processing Multiple Elements in an Array
3.5.5. Clearing/Deleting Arrays and Array Elements
3.5.6. Using Arrays in Conditional Statements
3.5.7. Computing for Statistical Aggregates
3.6. Tapsets
4. Useful SystemTap Scripts
4.1. Network
4.1.1. Network Profiling
4.1.2. Tracing Functions Called in Network Socket Code
4.1.3. Monitoring Incoming TCP Connections
4.1.4. Monitoring Network Packets Drops in Kernel
4.2. Disk
4.2.1. Summarizing Disk Read/Write Traffic
4.2.2. Tracking I/O Time For Each File Read or Write
4.2.3. Track Cumulative IO
4.2.4. I/O Monitoring (By Device)
4.2.5. Monitoring Reads and Writes to a File
4.2.6. Monitoring Changes to File Attributes
4.3. Profiling
4.3.1. Counting Function Calls Made
4.3.2. Call Graph Tracing
4.3.3. Determining Time Spent in Kernel and User Space
4.3.4. Monitoring Polling Applications
4.3.5. Tracking Most Frequently Used System Calls
4.3.6. Tracking System Call Volume Per Process
4.4. Identifying Contended User-Space Locks
5. Understanding SystemTap Errors
5.1. Parse and Semantic Errors
5.2. Run Time Errors and Warnings
6. References
A. Revision History