Chapter 26. Monitoring processes for performance bottlenecks using perf circular buffers
You can create circular buffers that take event-specific snapshots of data with the
perf tool in order to monitor performance bottlenecks in specific processes or parts of applications running on your system. In such cases,
perf only writes data to a
perf.data file for later analysis if a specified event is detected.
26.1. Circular buffers and event-specific snapshots with perf
When investigating performance issues in a process or application with
perf, it may not be affordable or appropriate to record data for hours preceding a specific event of interest occurring. In such cases, you can use
perf record to create custom circular buffers that take snapshots after specific events.
--overwrite option makes
perf record store all data in an overwritable circular buffer. When the buffer gets full,
perf record automatically overwrites the oldest records which, therefore, never get written to a
--switch-output-event options together configures a circular buffer that records and dumps data continuously until it detects the
--switch-output-event trigger event. The trigger event signals to
perf record that something of interest to the user has occurred and to write the data in the circular buffer to a
perf.data file. This collects specific data you are interested in while simultaneously reducing the overhead of the running
perf process by not writing data you do not want to a
26.2. Collecting specific data to monitor for performance bottlenecks using perf circular buffers
perf tool, you can create circular buffers that are triggered by events you specify in order to only collect data you are interested in. To create circular buffers that collect event-specific data, use the
--switch-output-event options for
You have the
perfuser space tool installed as described in Installing perf.
You have placed a uprobe in the process or application you are interested in monitoring at a location of interest within the process or application:
# perf probe -x /path/to/executable -a function Added new event: probe_executable:function (on function in /path/to/executable) You can now use it in all perf tools, such as: perf record -e probe_executable:function -aR sleep 1
Create the circular buffer with the uprobe as the trigger event:
# perf record --overwrite -e cycles --switch-output-event probe_executable:function ./executable [ perf record: dump data: Woken up 1 times ] [ perf record: Dump perf.data.2021021012231959 ] [ perf record: dump data: Woken up 1 times ] [ perf record: Dump perf.data.2021021012232008 ] ^C[ perf record: dump data: Woken up 1 times ] [ perf record: Dump perf.data.2021021012232082 ] [ perf record: Captured and wrote 5.621 MB perf.data.<timestamp> ]
This example initiates the executable and collects cpu cycles, specified after the
perfdetects the uprobe, the trigger event specified after the
--switch-output-eventoption. At that point,
perftakes a snapshot of all the data in the circular buffer and stores it in a unique
perf.datafile identified by timestamp. This example produced a total of 2 snapshots, the last
perf.datafile was forced by pressing Ctrl+c.