3.7. Application Profilers

Profiling is the process of gathering information about a program's behavior as it executes. You profile an application to determine which areas of a program can be optimized to increase the program's overall speed, reduce its memory usage, etc. Application profiling tools help to simplify this process.
There are three supported profiling tools for use with Red Hat Enterprise Linux 6: SystemTap, OProfile and Valgrind. Documenting these profiling tools is outside the scope of this guide; however, this section does provide links to further information and a brief overview of the tasks for which each profiler is suitable.

3.7.1. SystemTap

SystemTap is a tracing and probing tool that lets users monitor and analyze operating system activities (particularly kernel activities) in fine detail. It provides information similar to the output of tools like netstat, top, ps and iostat, but includes additional filtering and analysis options for the information that is collected.
SystemTap provides a deeper, more precise analysis of system activities and application behavior to allow you to pinpoint system and application bottlenecks.
The Function Callgraph plug-in for Eclipse uses SystemTap as a back-end, allowing it to thoroughly monitor the status of a program, including function calls, returns, times, and user-space variables, and display the information visually for easy optimization.
The Red Hat Enterprise Linux 7 SystemTap Beginner's Guide includes several sample scripts that are useful for profiling and monitoring performance. By default they are installed to the /usr/share/doc/systemtap-client-version/examples directory.

Network monitoring scripts (in examples/network)

nettop.stp
Every 5 seconds, prints a list of processes (process identifier and command) with the number of packets sent and received and the amount of data sent and received by the process during that interval.
socket-trace.stp
Instruments each of the functions in the Linux kernel's net/socket.c file, and prints trace data.
tcp_connections.stp
Prints information for each new incoming TCP connection accepted by the system. The information includes the UID, the command accepting the connection, the process identifier of the command, the port the connection is on, and the IP address of the originator of the request.
dropwatch.stp
Every 5 seconds, prints the number of socket buffers freed at locations in the kernel. Use the --all-modules option to see symbolic names.

Storage monitoring scripts (in examples/io)

disktop.stp
Checks the status of reading/writing disk every 5 seconds and outputs the top ten entries during that period.
iotime.stp
Prints the amount of time spent on read and write operations, and the number of bytes read and written.
traceio.stp
Prints the top ten executables based on cumulative I/O traffic observed, every second.
traceio2.stp
Prints the executable name and process identifier as reads and writes to the specified device occur.
inodewatch.stp
Prints the executable name and process identifier each time a read or write occurs to the specified inode on the specified major/minor device.
inodewatch2.stp
Prints the executable name, process identifier, and attributes each time the attributes are changed on the specified inode on the specified major/minor device.
The latencytap.stp script records the effect that different types of latency have on one or more processes. It prints a list of latency types every 30 seconds, sorted in descending order by the total time the process or processes spent waiting. This can be useful for identifying the cause of both storage and network latency. Red Hat recommends using the --all-modules option with this script to better enable the mapping of latency events. By default, this script is installed to the /usr/share/doc/systemtap-client-version/examples/profiling directory.
For further information about SystemTap, refer to the SystemTap Beginners Guide, available from http://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/.