24.6. Analyzing the Data

The same OProfile post-processing tools are used whether you collect your profile with operf or opcontrol in legacy mode.
By default, operf stores the profiling data in the current_dir/oprofile_data/ directory. You can change to a different location with the --session-dir option. The usual post-profiling analysis tools such as opreport and opannotate can be used to generate profile reports. These tools search for samples in current_dir/oprofile_data/ first. If this directory does not exist, the analysis tools use the standard session directory of /var/lib/oprofile/. Statistics, such as total samples received and lost samples, are written to the session_dir/samples/operf.log file.
When using legacy mode, the OProfile daemon, oprofiled, periodically collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:
opcontrol --dump
Each sample file name is based on the name of the executable. For example, the samples for the default event on a Pentium III processor for /bin/bash becomes:
\{root\}/bin/bash/\{dep\}/\{root\}/bin/bash/CPU_CLK_UNHALTED.100000
The following tools are available to profile the sample data once it has been collected:
  • opreport
  • opannotate
Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.

Warning

The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, back up the executable used to create the samples as well as the sample files. Note that the names of the sample file and the binary have to agree. You cannot make a backup if these names do not match. As an alternative, oparchive can be used to address this problem.
Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. The OProfile analysis tools use the executable file that created the samples during analysis. If the executable changes, the analysis tools will be unable to analyze the associated samples. See Section 24.5, “Saving Data in Legacy Mode” for details on how to back up the sample file.

24.6.1. Using opreport

The opreport tool provides an overview of all the executables being profiled. The following is part of a sample output from the opreport command:
~]$ opreport
Profiling through timer interrupt
TIMER:0|
samples|      %|
------------------
25926 97.5212 no-vmlinux
359  1.3504 pi
65  0.2445 Xorg
62  0.2332 libvte.so.4.4.0
56  0.2106 libc-2.3.4.so
34  0.1279 libglib-2.0.so.0.400.7
19  0.0715 libXft.so.2.1.2
17  0.0639 bash
8  0.0301 ld-2.3.4.so
8  0.0301 libgdk-x11-2.0.so.0.400.13
6  0.0226 libgobject-2.0.so.0.400.7
5  0.0188 oprofiled
4  0.0150 libpthread-2.3.4.so
4  0.0150 libgtk-x11-2.0.so.0.400.13
3  0.0113 libXrender.so.1.2.2
3  0.0113 du
1  0.0038 libcrypto.so.0.9.7a
1  0.0038 libpam.so.0.77
1  0.0038 libtermcap.so.2.0.8
1  0.0038 libX11.so.6.2
1  0.0038 libgthread-2.0.so.0.400.7
1  0.0038 libwnck-1.so.4.9.0
Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is the name of the executable.
See the opreport(1) manual page for a list of available command-line options, such as the -r option used to sort the output from the executable with the smallest number of samples to the one with the largest number of samples. You can also use the -t or --threshold option to trim the output of opcontrol.

24.6.2. Using opreport on a Single Executable

To retrieve more detailed profiled information about a specific executable, use:
opreport mode executable
Replace executable with the full path to the executable to be analyzed. mode stands for one of the following options:
-l
This option is used to list sample data by symbols. For example, running this command:
~]# opreport -l /usr/lib/tls/libc-version.so
produces the following output:
samples % symbol name 
12 21.4286 __gconv_transform_utf8_internal 
5 8.9286 _int_malloc 4 7.1429 malloc 
3 5.3571 __i686.get_pc_thunk.bx 
3 5.3571 _dl_mcount_wrapper_check 
3 5.3571 mbrtowc 
3 5.3571 memcpy 
2 3.5714 _int_realloc 
2 3.5714 _nl_intern_locale_data 
2 3.5714 free 
2 3.5714 strcmp 
1 1.7857 __ctype_get_mb_cur_max 
1 1.7857 __unregister_atfork 
1 1.7857 __write_nocancel 
1 1.7857 _dl_addr 
1 1.7857 _int_free 
1 1.7857 _itoa_word 
1 1.7857 calc_eclosure_iter 
1 1.7857 fopen@@GLIBC_2.1 
1 1.7857 getpid 
1 1.7857 memmove 
1 1.7857 msort_with_tmp 
1 1.7857 strcpy 
1 1.7857 strlen 
1 1.7857 vfprintf 
1 1.7857 write
The first column is the number of samples for the symbol, the second column is the percentage of samples for this symbol relative to the overall samples for the executable, and the third column is the symbol name.
To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.
-i symbol-name
List sample data specific to a symbol name. For example, running:
~]# opreport -l -i __gconv_transform_utf8_internal /usr/lib/tls/libc-version.so
returns the following output:
samples % symbol name 
12 100.000 __gconv_transform_utf8_internal
The first line is a summary for the symbol/executable combination.
The first column is the number of samples for the memory symbol. The second column is the percentage of samples for the memory address relative to the total number of samples for the symbol. The third column is the symbol name.
-d
This option lists sample data by symbols with more detail than the -l option. For example, with the following command:
~]# opreport -d -i __gconv_transform_utf8_internal /usr/lib/tls/libc-version.so
this output is returned:
vma samples % symbol name 
00a98640 12 100.000 __gconv_transform_utf8_internal 
00a98640 1 8.3333 
00a9868c 2 16.6667 
00a9869a 1 8.3333 
00a986c1 1 8.3333 
00a98720 1 8.3333 
00a98749 1 8.3333 
00a98753 1 8.3333 
00a98789 1 8.3333 
00a98864 1 8.3333 
00a98869 1 8.3333 
00a98b08 1 8.3333
The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.
-e symbol-name
With this option, you can exclude some symbols from the output. Replace symbol-name with the comma-separated list of symbols you want to exclude.
session:name
Here, you can specify the full path to the session, a directory relative to the /var/lib/oprofile/samples/ directory, or if you are using operf, a directory relative to ./oprofile_data/samples/.

24.6.3. Getting More Detailed Output on the Modules

OProfile collects data on a system-wide basis for kernel- and user-space code running on the machine. However, once a module is loaded into the kernel, the information about the origin of the kernel module is lost. The module could come from the initrd file on boot up, the directory with the various kernel modules, or a locally created kernel module. As a result, when OProfile records samples for a module, it just lists the samples for the modules for an executable in the root directory, but this is unlikely to be the place with the actual code for the module. You will need to take some steps to make sure that analysis tools get the proper executable.
To get a more detailed view of the actions of the module, you will need to either have the module "unstripped" (that is installed from a custom build) or have the debuginfo package installed for the kernel.
Find out which kernel is running with the uname -a command, obtain the appropriate debuginfo package and install it on the machine.
Then proceed with clearing out the samples from previous runs with the following command:
opcontrol --reset
To start the monitoring process, for example, on a machine with Westmere processor, run the following command:
~]# opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux --event=CPU_CLK_UNHALTED:500000
Then the detailed information, for instance, for the ext4 module can be obtained with:
~]# opreport /ext4 -l --image-path /usr/lib/modules/`uname -r`/kernel
CPU: Intel Westmere microarchitecture, speed 2.667e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 500000
warning: could not check that the binary file /lib/modules/2.6.32-191.el6.x86_64/kernel/fs/ext4/ext4.ko has not been modified since the profile was taken. Results may be inaccurate.
samples  %        symbol name
1622      9.8381  ext4_iget
1591      9.6500  ext4_find_entry
1231      7.4665  __ext4_get_inode_loc
783       4.7492  ext4_ext_get_blocks
752       4.5612  ext4_check_dir_entry
644       3.9061  ext4_mark_iloc_dirty
583       3.5361  ext4_get_blocks
583       3.5361  ext4_xattr_get
479       2.9053  ext4_htree_store_dirent
469       2.8447  ext4_get_group_desc
414       2.5111  ext4_dx_find_entry

24.6.4. Using opannotate

The opannotate tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting generated files should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.
For this utility to work, the appropriate debuginfo package for the executable must be installed on the system. On Red Hat Enterprise Linux, the debuginfo packages are not automatically installed with the corresponding packages that contain the executable. You have to obtain and install them separately.
The general syntax for opannotate is as follows:
opannotate --search-dirs src-dir --source executable
These command-line options are mandatory. Replace src-dir with a path to the directory containing the source code and specify the executable to be analyzed. See the opannotate(1) manual page for a list of additional command line options.