4.8. About Perf

Perf is a performance analysis tool. It provides a simple command line interface and separates the CPU hardware difference in Linux performance measurements. Perf is based on the perf_events interface exported by the kernel.
One advantage of perf is that it is both kernel and architecture neutral. The analysis data can be reviewed without requiring specific system configuration.
To be able to use perf, install the perf package by running the following command as root: 
~]# yum install perf
Perf has the following options. Examples of the most common options and features follow, but further information on all options are available with the perf help COMMAND.

Example 4.2. Example of perf Options

]# perf

 usage: perf [--version] [--help] COMMAND [ARGS]

 The most commonly used perf commands are:
   annotate        Read perf.data (created by perf record) and display annotated code
   archive         Create archive with object files with build-ids found in perf.data file
   bench           General framework for benchmark suites
   buildid-cache   Manage build-id cache.
   buildid-list    List the buildids in a perf.data file
   diff            Read two perf.data files and display the differential profile
   evlist          List the event names in a perf.data file
   inject          Filter to augment the events stream with additional information
   kmem            Tool to trace/measure kernel memory(slab) properties
   kvm             Tool to trace/measure kvm guest os
   list            List all symbolic event types
   lock            Analyze lock events
   record          Run a command and record its profile into perf.data
   report          Read perf.data (created by perf record) and display the profile
   sched           Tool to trace/measure scheduler properties (latencies)
   script          Read perf.data (created by perf record) and display trace output
   stat            Run a command and gather performance counter statistics
   test            Runs sanity tests.
   timechart       Tool to visualize total system behavior during a workload
   top             System profiling tool.
   trace           strace inspired tool
   probe           Define new dynamic tracepoints

See 'pert help COMMAND' for more information on a specific command.
These following examples show a selection of the most used features, including record, archive, report, stat and list.

Example 4.3. Perf Record

The perf record feature is used for collecting system-wide statistics. It can be used in all processors. 
~]# perf record -a
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.725 MB perf.data (~31655 samples) ]
In this example, all CPUs are denoted with the option -a, and the process was terminated after a few seconds. The results show that it collected 0.725 MB of data, and created the following file of results. 
~]# ls
perf.data

Example 4.4. Example of the Perf Report and Archive Features

The data from the perf record feature can now be directly investigated using the perf report commands. If the samples are to be analyzed on a different system, use the perf archive command. This will not always be necessary as the DSOs (such as binaries and libraries) may already be present in the analysis system, such as the ~/.debug/ cache or if both systems have the same set of binaries.
Run the archive command to create an archive of results. 
~]# perf archive
Collect the results as a tar archive to prepare the data for the pref report. 
~]# tar xvf perf.data.tar.bz2 -C ~/.debug
Run the perf report to analyze the tarball. 
~]# perf report
The output of the report is sorted according to the maximum CPU usage in percentage by the application. It shows if the sample has occurred in kernel or user space of the process.
A kernel sample, if not taking place in a kernel module will be marked by the notation [kernel.kallsyms]. If a kernel sample is taking place in the kernel module, it will be marked as [module], [ext4]. For a process in user space, the results might show the shared library linked with the process.
The report denotes whether the process also occurs in kernel or user space. The result [.] indicates user space and [k] indicates kernel space. Finer grained details are available for review, including data appropriate for experienced perf developers.

Example 4.5. Example of the Perf List and Stat Features

The perf list and stat features show all the hardware or software trace points that can be probed.
The following example shows how to view the number of context switches with the perf stat feature. 
~]# perf stat -e context-switches -a sleep 5
Performance counter stats for 'sleep 5':

            15,619 context-switches

       5.002060064 seconds time elapsed
The results show that in 5 seconds, 15619 context switches took place. Filesystem activity is also viewable, as shown in the following example script. 
~]# for i in {1..100}; do touch /tmp/$i; sleep 1; done
In another terminal, run the following perf stat feature. 
~]# perf stat -e ext4:ext4_request_inode -a sleep 5
 Performance counter stats for 'sleep 5':

                 5 ext4:ext4_request_inode

       5.002253620 seconds time elapsed
The results show that in 5 seconds the script asked to create 5 files, indicating that there are 5 inode requests.
There are a range of available options to get the hardware tracepoint activity. The following example shows a selection of the options in the perf list feature. 
List of pre-defined events (to be used in -e):
  cpu-cycles OR cycles                               [Hardware event]
  stalled-cycles-frontend OR idle-cycles-frontend    [Hardware event]
  stalled-cycles-backend OR idle-cycles-backend      [Hardware event]
  instructions                                       [Hardware event]
  cache-references                                   [Hardware event]
  cache-misses                                       [Hardware event]
  branch-instructions OR branches                    [Hardware event]
  branch-misses                                      [Hardware event]
  bus-cycles                                         [Hardware event]

  cpu-clock                                          [Software event]
  task-clock                                         [Software event]
  page-faults OR faults                              [Software event]
  minor-faults                                       [Software event]
  major-faults                                       [Software event]
  context-switches OR cs                             [Software event]
  cpu-migrations OR migrations                       [Software event]
  alignment-faults                                   [Software event]
  emulation-faults                                   [Software event]
  ...[output truncated]...

Important

Sampling at too high a frequency can negatively impact the performance of your real-time system.