Chapter 16. Getting started with flamegraphs

As a system administrator, you can use flamegraphs to to create visualizations of system performance data recorded with the perf tool. As a software developer, you can use flamegraphs to create visualizations of application performance data recorded with the perf tool.

Sampling stack traces is a common technique for profiling CPU performance with the perf tool. Unfortunately, the results of profiling stack traces with perf can be extremely verbose and labor-intensive to analyze. flamegraphs are visualizations created from data recorded with perf to make identifying hot code-paths faster and easier.

16.1. Installing flamegraphs

To begin using flamegraphs, install the required package:

Procedure

  • Install the flamegraphs package:

    # yum install js-d3-flame-graph

16.2. Creating flamegraphs over the entire system

This procedure describes how to visualize performance data recorded over an entire system using flamegraphs.

Prerequisites

Procedure

  • Record the data and create the visualization:

    # perf script flamegraph -a -F 99 sleep 60

    This command samples and records performance data over the entire system for 60 seconds, as stipulated by use of the sleep command, and then constructs the visualization which will be stored in the current active directory as flamegraph.html. The command will sample call-graph data by default and takes the same arguments as the perf tool, in this particular case:

    -a
    Stipulates to record data over the entire system.
    -F
    To set the sampling frequency per second.

Verification steps

  • For analysis, view the generated flamegraph:

    # xdg-open flamegraph.html

    This command opens the flamegraph in the default browser:

flamegraph allcpus

16.3. Creating flamegraphs over specific processes

You can use flamegraphs to visualize performance data recorded over specific running processes.

Prerequisites

Procedure

  • Record the data and create the visualization:

    # perf script flamegraph -a -F 99 -p ID1,ID2 sleep 60

    This command samples and records performance data of the processes with the process ID’s ID1 and ID2 for 60 seconds, as stipulated by use of the sleep command, and then constructs the visualization which will be stored in the current active directory as flamegraph.html. The command will sample call-graph data by default and takes the same arguments as the perf tool, in this particular case:

    -a
    Stipulates to record data over the entire system.
    -F
    To set the sampling frequency per second.
    -p
    To stipulate specific process ID’s to sample and record data over.

Verification steps

  • For analysis, view the generated flamegraph:

    # xdg-open flamegraph.html

    This previous command opens the flamegraph in the default browser:

flamegraph

16.4. Interpreting flamegraphs

Each box in the flamegraph represents a different function in the stack. The y-axis shows the depth of the stack with the topmost box in each stack being the function that was actually on-CPU and everything below it being ancestry. The x-axis displays the population of the sampled call-graph data. The children of a stack in a given row are displayed based on the number of samples taken of each respective function in descending order along the x-axis; the x-axis does not represent the passing of time. The wider an individual box is, the more frequent it was on-CPU or part of an on-CPU ancestry at the time the data was being sampled.

IMPORTANT
Boxes representing user-space functions may be labeled as Unknown in flamegraphs because the binary of the function is stripped. The debuginfo package of the executable must be installed or, if the executable is a locally developed application, the application must be compiled with debugging information, the -g option in GCC, to display the function names or symbols in such a situation.

flamegraph

Procedure

  • To reveal the names of functions which may have not been displayed previously and further investigate the data click on a box within the flamegraph to zoom into the stack at that given location:

zoomed in flamegraph

  • To return to the default view of the flamegraph, click the Reset Zoom button.