Chapter 13. Getting started with perf

As a system administrator, you can use the perf tool to collect and analyze performance data of your system.

13.1. Introduction to perf

The perf user-space tool interfaces with the kernel-based subsystem Performance Counters for Linux (PCL). perf is a powerful tool that uses the Performance Monitoring Unit (PMU) to measure, record, and monitor a variety of hardware and software events. perf also supports tracepoints, kprobes, and uprobes.

13.2. Installing perf

This procedure installs the perf user-space tool.

Procedure

  • Install the perf tool: 

    # yum install perf

13.3. Common Perf Commands

This section provides an overview of commonly used perf commands.

Commonly used perf commands

perf stat
This command provides overall statistics for common performance events, including instructions executed and clock cycles consumed. Options allow for selection of events other than the default measurement events.
perf record
This command records performance data into a file, perf.data, which can be later analyzed using the perf report command.
perf report
This command reads and displays the performance data from the perf.data file created by perf record.
perf list
This command lists the events available on a particular machine. These events will vary based on performance monitoring hardware and software configuration of the system.
perf top
This command performs a similar function to the top utility. It generates and displays a performance counter profile in realtime.
perf trace
This command performs a similar function to the strace tool. It monitors the system calls used by a specified thread or process and all signals received by that application.
perf help
This command displays a complete list of perf commands.

Additional resources

  • To list additional subcommand options of the subcommands and their descriptions, add the -h option to the target command.

13.4. Real time profiling of CPU usage with perf top

You can use the perf top command to measure CPU usage of different functions in real time.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

13.4.1. The purpose of perf top

The perf top command is used for real time system profiling and functions similarly to the top utility. However, where the top utility generally shows you how much CPU time a given process or thread is using, perf top shows you how much CPU time each specific function uses. In its default state, perf top tells you about functions being used across all CPUs in both the user-space and the kernel-space. To use perf top you need root access.

13.4.2. Profiling CPU usage with perf top

This procedure activates perf top and profiles CPU usage in real time.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.
  • You have root access

Procedure

  • Start the perf top monitoring interface: 

    # perf top

    Example 13.1. Perf top output

    --------------------------------------------------------------------
PerfTop:   20806 irqs/sec  kernel:57.3%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 8 CPUs)
---------------------------------------------------------------------
Overhead  Shared Object       Symbol
   2.20%  [kernel]            [k] do_syscall_64
   2.17%  [kernel]            [k] module_get_kallsym
   1.49%  [kernel]            [k] copy_user_enhanced_fast_string
   1.37%  libpthread-2.29.so  [.] __pthread_mutex_lock
   1.31%  [unknown]           [.] 0000000000000000
   1.07%  [kernel]            [k] psi_task_change
   1.04%  [kernel]            [k] switch_mm_irqs_off
   0.94%  [kernel]            [k] __fget
   0.74%  [kernel]            [k] entry_SYSCALL_64
   0.69%  [kernel]            [k] syscall_return_via_sysret
   0.69%  libxul.so           [.] 0x000000000113f9b0
   0.67%  [kernel]            [k] kallsyms_expand_symbol.constprop.0
   0.65%  firefox             [.] moz_xmalloc
   0.65%  libpthread-2.29.so  [.] __pthread_mutex_unlock_usercnt
   0.60%  firefox             [.] free
   0.60%  libxul.so           [.] 0x000000000241d1cd
   0.60%  [kernel]            [k] do_sys_poll
   0.58%  [kernel]            [k] menu_select
   0.56%  [kernel]            [k] _raw_spin_lock_irqsave
   0.55%  perf                [.] 0x00000000002ae0f3

    In the previous example, the kernel function do_syscall_64 is using the most CPU time.

Additional resources

  • The perf-top(1) man page.

13.4.3. Interpretation of perf top output

The "Overhead" column
Displays the percent of CPU a given function is using.
The "Shared Object" column
Displays name of the program or library which is using the function.
The "Symbol" column
Displays the function name or symbol. Functions executed in the kernel-space are identified by [k] and functions executed in the user-space are identified by [.].

13.4.4. Why perf displays some function names as raw function addresses

For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped.

The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information turned on (the -g option in GCC) to display the function names or symbols in such a situation.

Additional Resources

13.4.5. Enabling debug and source repositories

A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance.

Procedure

  • Enable the source and debug information package channels: 

    # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-debug-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-source-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-debug-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-source-rpms

    The $(uname -i) part is automatically replaced with a matching value for architecture of your system:

    Architecture nameValue

    64-bit Intel and AMD

    x86_64

    64-bit ARM

    aarch64

    IBM POWER

    ppc64le

    64-bit IBM Z

    s390x

13.4.6. Getting debuginfo packages for an application or library using GDB

Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package.

Prerequisites

  • The application or library you want to debug must be installed on the system.
  • GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications.
  • Channels providing debuginfo and debugsource packages must be configured and enabled on the system.

Procedure

  1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. 

    $ gdb -q /bin/ls
Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.30-6.el8.x86_64
(gdb)

  2. Exit GDB: type q and confirm with Enter. 

    (gdb) q

  3. Run the command suggested by GDB to install the required debuginfo packages: 

    # dnf debuginfo-install coreutils-8.30-6.el8.x86_64

    The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files.

  4. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

Additional resources

13.5. Counting events during process execution

You can use the perf stat command to count hardware and software events during process execution.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

13.5.1. The purpose of perf stat

The perf stat command executes a specified command, keeps a running count of hardware and software event occurrences during the commands execution, and generates statistics of these counts. If you do not specify any events, then perf stat counts a set of common hardware and software events.

13.5.2. Counting events with perf stat

You can use perf stat to count hardware and software event occurrences during command execution and generate statistics of these counts. By default, perf stat operates in per-thread mode.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

Procedure

  • Count the events.

    • Running the perf stat command without root access will only count events occurring in the user space: 

      $ perf stat ls

      Example 13.2. Output of perf stat ran without root access

      Desktop  Documents  Downloads  Music  Pictures  Public  Templates  Videos

 Performance counter stats for 'ls':

              1.28 msec task-clock:u               #    0.165 CPUs utilized
                 0      context-switches:u         #    0.000 M/sec
                 0      cpu-migrations:u           #    0.000 K/sec
               104      page-faults:u              #    0.081 M/sec
         1,054,302      cycles:u                   #    0.823 GHz
         1,136,989      instructions:u             #    1.08  insn per cycle
           228,531      branches:u                 #  178.447 M/sec
            11,331      branch-misses:u            #    4.96% of all branches

       0.007754312 seconds time elapsed

       0.000000000 seconds user
       0.007717000 seconds sys

      As you can see in the previous example, when perf stat runs without root access the event names are followed by :u, indicating that these events were counted only in the user-space.

    • To count both user-space and kernel-space events, you must have root access when running perf stat: 

      # perf stat ls

      Example 13.3. Output of perf stat ran with root access

      Desktop  Documents  Downloads  Music  Pictures  Public  Templates  Videos

 Performance counter stats for 'ls':

              3.09 msec task-clock                #    0.119 CPUs utilized
                18      context-switches          #    0.006 M/sec
                 3      cpu-migrations            #    0.969 K/sec
               108      page-faults               #    0.035 M/sec
         6,576,004      cycles                    #    2.125 GHz
         5,694,223      instructions              #    0.87  insn per cycle
         1,092,372      branches                  #  352.960 M/sec
            31,515      branch-misses             #    2.89% of all branches

       0.026020043 seconds time elapsed

       0.000000000 seconds user
       0.014061000 seconds sys

      • By default, perf stat operates in per-thread mode. To change to CPU-wide event counting, pass the -a option to perf stat. To count CPU-wide events, you need root access: 

        # perf stat -a ls

Additional resources

  • The perf-stat(1) man page.

13.5.3. Interpretation of perf stat output

perf stat executes a specified command and counts event occurrences during the commands execution and displays statistics of these counts in three columns:

  1. The number of occurrences counted for a given event
  2. The name of the event that was counted

  3. When related metrics are available, a ratio or percentage is displayed after the hash sign (#) in the right-most column.

    • For example, when running in default mode, perf stat counts both cycles and instructions and, therefore, calculates and displays instructions per cycle in the right-most column. You can see similar behavior with regard to branch-misses as a percent of all branches since both events are counted by default.

13.5.4. Attaching perf stat to a running process

You can attach perf stat to a running process. This will instruct perf stat to count event occurrences only in the specified processes during the execution of a command.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

Procedure

  • Attach perf stat to a running process: 

    $ perf stat -p ID1,ID2 sleep seconds

    The previous example counts events in the processes with the IDs of ID1 and ID2 for a time period of seconds seconds as dictated by using the sleep command.

Additional resources

  • The perf-stat(1) man page.

13.6. Recording and analyzing performance profiles with perf

The perf tool allows you to record performance data and analyze it at a later time.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

13.6.1. The purpose of perf record

The perf record command samples performance data and stores it in a file, perf.data, which can be read and visualized with other perf commands. perf.data is generated in the current directory and can be accessed at a later time, possibly on a different machine.

If you do not specify a command for perf record to record during, it will record until you manually stop the process by pressing Ctrl+C. You can attach perf record to specific processes by passing the -p option followed by one or more process IDs. You can run perf record without root access, however, doing so will only sample performance data in the user space. In the default mode, perf record uses CPU cycles as the sampling event and operates in per-thread mode with inherit mode enabled.

13.6.2. Recording a performance profile without root access

You can use perf record without root access to sample and record performance data in the user-space only.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

Procedure

  • Sample and record the performance data: 

    $ perf record command

    Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additonal resources

  • The perf-record(1) man page.

13.6.3. Recording a performance profile with root access

You can use perf record with root access to sample and record performance data in both the user-space and the kernel-space simultaneously.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.
  • You have root access.

Procedure

  • Sample and record the performance data: 

    # perf record command

    Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additonal resources

  • The perf-record(1) man page.

13.6.4. Recording a performance profile in per-CPU mode

You can use perf record in per-CPU mode to sample and record performance data in both and user-space and the kernel-space simultaneously across all threads on a monitored CPU. By default, per-CPU mode monitors all online CPUs.

Prerequisites

  • The perf user space tool is installed. For more information, see Installing perf.

Procedure

  • Sample and record the performance data: 

    # perf record -a command

    Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

Additional resources

  • The perf-record(1) man page.

13.6.5. Capturing call graph data with perf record

You can configure the perf record tool so that it records which function is calling other functions in the performance profile. This helps to identify a bottleneck if several processes are calling the same function.

Prerequisites

  • The perf user space tool is installed. For more information, see Installing perf.

Procedure

  • Sample and record performance data with the --call-graph option: 

    $ perf record --call-graph method command
    • Replace command with the command you want to sample data during. If you do not specify a command, then perf record will sample data until you manually stop it by pressing Ctrl+C.

    • Replace method with one of the following unwinding methods:

      fp
      Uses the frame pointer method. Depending on compiler optimization, such as with binaries built with the GCC option --fomit-frame-pointer, this may not be able to unwind the stack.
      dwarf
      Uses DWARF Call Frame Information to unwind the stack.
      lbr
      Uses the last branch record hardware on Intel processors.

Additional resources

  • The perf-record(1) man page.

13.6.6. Analyzing perf.data with perf report

You can use perf report to display and analyze a perf.data file.

Prerequisites

  • The perf user space tool is installed. For more information, see Installing perf.
  • There is a perf.data file in the current directory.
  • If the perf.data file was created with root access, you need to run perf report with root access too.

Procedure

  • Display the contents of the perf.data file for further analysis: 

    # perf report

    Example 13.4. Example output

    Samples: 2K of event 'cycles', Event count (approx.): 235462960
Overhead  Command          Shared Object                     Symbol
   2.36%  kswapd0          [kernel.kallsyms]                 [k] page_vma_mapped_walk
   2.13%  sssd_kcm         libc-2.28.so                      [.] __memset_avx2_erms
   2.13%  perf             [kernel.kallsyms]                 [k] smp_call_function_single
   1.53%  gnome-shell      libc-2.28.so                      [.] __strcmp_avx2
   1.17%  gnome-shell      libglib-2.0.so.0.5600.4           [.] g_hash_table_lookup
   0.93%  Xorg             libc-2.28.so                      [.] __memmove_avx_unaligned_erms
   0.89%  gnome-shell      libgobject-2.0.so.0.5600.4        [.] g_object_unref
   0.87%  kswapd0          [kernel.kallsyms]                 [k] page_referenced_one
   0.86%  gnome-shell      libc-2.28.so                      [.] __memmove_avx_unaligned_erms
   0.83%  Xorg             [kernel.kallsyms]                 [k] alloc_vmap_area
   0.63%  gnome-shell      libglib-2.0.so.0.5600.4           [.] g_slice_alloc
   0.53%  gnome-shell      libgirepository-1.0.so.1.0.0      [.] g_base_info_unref
   0.53%  gnome-shell      ld-2.28.so                        [.] _dl_find_dso_for_object
   0.49%  kswapd0          [kernel.kallsyms]                 [k] vma_interval_tree_iter_next
   0.48%  gnome-shell      libpthread-2.28.so                [.] __pthread_getspecific
   0.47%  gnome-shell      libgirepository-1.0.so.1.0.0      [.] 0x0000000000013b1d
   0.45%  gnome-shell      libglib-2.0.so.0.5600.4           [.] g_slice_free1
   0.45%  gnome-shell      libgobject-2.0.so.0.5600.4        [.] g_type_check_instance_is_fundamentally_a
   0.44%  gnome-shell      libc-2.28.so                      [.] malloc
   0.41%  swapper          [kernel.kallsyms]                 [k] apic_timer_interrupt
   0.40%  gnome-shell      ld-2.28.so                        [.] _dl_lookup_symbol_x
   0.39%  kswapd0          [kernel.kallsyms]                 [k] __raw_callee_save___pv_queued_spin_unlock

Additonal resources

  • The perf-report(1) man page.

13.6.7. Interpretation of perf report output

The table displayed by running the perf report command sorts the data into several columns:

The 'Overhead' column
Indicates what percentage of overall samples were collected in that particular function.
The 'Command' column
Tells you which process the samples were collected from.
The 'Shared Object' column
Displays the name of the ELF image where the samples come from (the name [kernel.kallsyms] is used when the samples come from the kernel).
The 'Symbol' column
Displays the function name or symbol.

In default mode, the functions are sorted in descending order with those with the highest overhead displayed first.

13.6.8. Why perf displays some function names as raw function addresses

For kernel functions, perf uses the information from the /proc/kallsyms file to map the samples to their respective function names or symbols. For functions executed in the user space, however, you might see raw function addresses because the binary is stripped.

The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information turned on (the -g option in GCC) to display the function names or symbols in such a situation.

Additional Resources

Note

It is not necessary to re-run perf record after installing the debuginfo associated with an executable. Simply re-run perf report.

13.6.9. Enabling debug and source repositories

A standard installation of Red Hat Enterprise Linux does not enable the debug and source repositories. These repositories contain information needed to debug the system components and measure their performance.

Procedure

  • Enable the source and debug information package channels: 

    # subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-debug-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-baseos-source-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-debug-rpms
# subscription-manager repos --enable rhel-8-for-$(uname -i)-appstream-source-rpms

    The $(uname -i) part is automatically replaced with a matching value for architecture of your system:

    Architecture nameValue

    64-bit Intel and AMD

    x86_64

    64-bit ARM

    aarch64

    IBM POWER

    ppc64le

    64-bit IBM Z

    s390x

13.6.10. Getting debuginfo packages for an application or library using GDB

Debugging information is required to debug code. For code that is installed from a package, the GNU Debugger (GDB) automatically recognizes missing debug information, resolves the package name and provides concrete advice on how to get the package.

Prerequisites

  • The application or library you want to debug must be installed on the system.
  • GDB and the debuginfo-install tool must be installed on the system. For details, see Setting up to debug applications.
  • Channels providing debuginfo and debugsource packages must be configured and enabled on the system.

Procedure

  1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run. 

    $ gdb -q /bin/ls
Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.30-6.el8.x86_64
(gdb)

  2. Exit GDB: type q and confirm with Enter. 

    (gdb) q

  3. Run the command suggested by GDB to install the required debuginfo packages: 

    # dnf debuginfo-install coreutils-8.30-6.el8.x86_64

    The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files.

  4. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

Additional resources