Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 28. Monitoring processes for performance bottlenecks using perf circular buffers

You can create circular buffers that take event-specific snapshots of data with the perf tool in order to monitor performance bottlenecks in specific processes or parts of applications running on your system. In such cases, perf only writes data to a perf.data file for later analysis if a specified event is detected.

28.1. Circular buffers and event-specific snapshots with perf

When investigating performance issues in a process or application with perf, it may not be affordable or appropriate to record data for hours preceding a specific event of interest occurring. In such cases, you can use perf record to create custom circular buffers that take snapshots after specific events.

The --overwrite option makes perf record store all data in an overwritable circular buffer. When the buffer gets full, perf record automatically overwrites the oldest records which, therefore, never get written to a perf.data file.

Using the --overwrite and --switch-output-event options together configures a circular buffer that records and dumps data continuously until it detects the --switch-output-event trigger event. The trigger event signals to perf record that something of interest to the user has occurred and to write the data in the circular buffer to a perf.data file. This collects specific data you are interested in while simultaneously reducing the overhead of the running perf process by not writing data you do not want to a perf.data file.

28.2. Collecting specific data to monitor for performance bottlenecks using perf circular buffers

With the perf tool, you can create circular buffers that are triggered by events you specify in order to only collect data you are interested in. To create circular buffers that collect event-specific data, use the --overwrite and --switch-output-event options for perf.

Prerequisites

  • You have the perf user space tool installed as described in Installing perf.

  • You have placed a uprobe in the process or application you are interested in monitoring at a location of interest within the process or application: 

    # perf probe -x /path/to/executable -a function
Added new event:
  probe_executable:function   (on function in /path/to/executable)

You can now use it in all perf tools, such as:

        perf record -e probe_executable:function -aR sleep 1

Procedure

  • Create the circular buffer with the uprobe as the trigger event: 

    # perf record --overwrite -e cycles --switch-output-event probe_executable:function ./executable
[ perf record: dump data: Woken up 1 times ]
[ perf record: Dump perf.data.2021021012231959 ]
[ perf record: dump data: Woken up 1 times ]
[ perf record: Dump perf.data.2021021012232008 ]
^C[ perf record: dump data: Woken up 1 times ]
[ perf record: Dump perf.data.2021021012232082 ]
[ perf record: Captured and wrote 5.621 MB perf.data.<timestamp> ]

    This example initiates the executable and collects cpu cycles, specified after the -e option, until perf detects the uprobe, the trigger event specified after the --switch-output-event option. At that point, perf takes a snapshot of all the data in the circular buffer and stores it in a unique perf.data file identified by timestamp. This example produced a total of 2 snapshots, the last perf.data file was forced by pressing Ctrl+c.