Chapter 28. Tracing latencies using ftrace
ftrace utility is one of the diagnostic facilities provided with the RHEL for Real Time kernel.
ftrace can be used by developers to analyze and debug latency and performance issues that occur outside of the user-space. The
ftrace utility has a variety of options that allow you to use the utility in different ways. It can be used to trace context switches, measure the time it takes for a high-priority task to wake up, the length of time interrupts are disabled, or list all the kernel functions executed during a given period.
Some of the
ftrace tracers, such as the function tracer, can produce exceedingly large amounts of data, which can turn trace log analysis into a time-consuming task. However, you can instruct the tracer to begin and end only when the application reaches critical code paths.
- You have administrator privileges.
28.1. Using the ftrace utility to trace latencies
You can trace latencies using the
View the available tracers on the system.
# cat /sys/kernel/debug/tracing/available_tracers function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop
The user interface for
ftraceis a series of files within
ftracefiles are also located in the
Move to the
# cd /sys/kernel/debug/tracing
The files in this directory can only be modified by the root user, because enabling tracing can have an impact on the performance of the system.
To start a tracing session:
Select a tracer you want to use from the list of available tracers in
Insert the name of the selector into the
# echo preemptoff > /sys/kernel/debug/tracing/current_tracerNote
If you use a single '>' with the echo command, it will override any existing value in the file. If you wish to append the value to the file, use '>>' instead.
- Select a tracer you want to use from the list of available tracers in
The function-trace option is useful because tracing latencies with
preemptirqsoff, and so on automatically enables
function tracing, which may exaggerate the overhead.
function_graphtracing are enabled:
# cat /sys/kernel/debug/tracing/options/function-trace 1
A value of
function_graphtracing are enabled.
A value of
function_graphtracing are disabled.
- A value of
function_graphtracing are enabled. To turn
function_graphtracing on or off, echo the appropriate value to the
# echo 0 > /sys/kernel/debug/tracing/options/function-trace # echo 1 > /sys/kernel/debug/tracing/options/function-traceImportant
When using the
echocommand, ensure you place a space character in between the value and the
>character. At the shell prompt, using
2>(without a space character) refers to standard input, standard output, and standard error. Using them by mistake could result in an unexpected trace output.
Adjust the details and parameters of the tracers by changing the values for the various files in the
wakeuptracers continuously monitor latencies. When they record a latency greater than the one recorded in
tracing_max_latencythe trace of that latency is recorded, and
tracing_max_latencyis updated to the new maximum time. In this way,
tracing_max_latencyalways shows the highest recorded latency since it was last reset.
To reset the maximum latency, echo
# echo 0 > /sys/kernel/debug/tracing/tracing_max_latency
To see only latencies greater than a set amount, echo the amount in microseconds:
# echo 200 > /sys/kernel/debug/tracing/tracing_max_latency
When the tracing threshold is set, it overrides the maximum latency setting. When a latency is recorded that is greater than the threshold, it will be recorded regardless of the maximum latency. When reviewing the trace file, only the last recorded latency is shown.
To set the threshold, echo the number of microseconds above which latencies must be recorded:
# echo 200 > /sys/kernel/debug/tracing/tracing_thresh
View the trace logs:
# cat /sys/kernel/debug/tracing/trace
To store the trace logs, copy them to another file:
# cat /sys/kernel/debug/tracing/trace > /tmp/lat_trace_log
View the functions being traced:
# cat /sys/kernel/debug/tracing/set_ftrace_filter
Filter the functions being traced by editing the settings in
/sys/kernel/debug/tracing/set_ftrace_filter. If no filters are specified in the file, all functions are traced.
To change filter settings, echo the name of the function to be traced. The filter allows the use of a '*' wildcard at the beginning or end of a search term.
For examples, see ftrace examples.
28.2. ftrace files
The following are the main files in the
The file that shows the output of an
ftracetrace. This is really a snapshot of the trace in time, because the trace stops when this file is read, and it does not consume the events read. That is, if the user disabled tracing and reads this file, it will report the same thing every time it is read.
The file that shows the output of an
ftracetrace as it reads the trace live. It is a producer/consumer trace. That is, each read will consume the event that is read. This can be used to read an active trace without stopping the trace as it is read.
- A list of ftrace tracers that have been compiled into the kernel.
Enables or disables an
- A directory that contains events to trace and can be used to enable or disable events, as well as set filters for the events.
Disable and enable recording to the
ftracebuffer. Disabling tracing via the
tracing_onfile does not disable the actual tracing that is happening inside the kernel. It only disables writing to the buffer. The work to do the trace still happens, but the data does not go anywhere.
28.3. ftrace tracers
Depending on how the kernel is configured, not all tracers may be available for a given kernel. For the RHEL for Real Time kernels, the trace and debug kernels have different tracers than the production kernel does. This is because some of the tracers have a noticeable overhead when the tracer is configured into the kernel, but not active. Those tracers are only enabled for the
- One of the most widely applicable tracers. Traces the function calls within the kernel. This can cause noticeable overhead depending on the number of functions traced. When not active, it creates little overhead.
function_graphtracer is designed to present results in a more visually appealing format. This tracer also traces the exit of the function, displaying a flow of function calls in the kernel.Note
This tracer has more overhead than the
functiontracer when enabled, but the same low overhead when disabled.
- A full CPU tracer that reports the activity happening across all CPUs. It records the time that it takes to wake up the highest priority task in the system, whether that task is a real time task or not. Recording the max time it takes to wake up a non-real time task hides the times it takes to wake up a real time task.
- A full CPU tracer that reports the activity happening across all CPUs. It records the time that it takes from the current highest priority task to wake up to until the time it is scheduled. This tracer only records the time for real time tasks.
- Traces the areas that disable preemption or interrupts, and records the maximum amount of time for which preemption or interrupts were disabled.
- Similar to the preemptirqsoff tracer, but traces only the maximum interval for which pre-emption was disabled.
- Similar to the preemptirqsoff tracer, but traces only the maximum interval for which interrupts were disabled.
The default tracer. It does not provide any tracing facility itself, but as events may interleave into any tracer, the
noptracer is used for specific interest in tracing events.
28.4. ftrace examples
The following provides a number of examples for changing the filtering of functions being traced. You can use the * wildcard at both the beginning and end of a word. For example:
*irq\* will select all functions that contain irq in the name. The wildcard cannot, however, be used inside a word.
Encasing the search term and the wildcard character in double quotation marks ensures that the shell will not attempt to expand the search to the present working directory.
Examples of filters
Trace only the
# echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
Trace all functions that end with
# echo "*lock" > /sys/kernel/debug/tracing/set_ftrace_filter
Trace all functions that start with
# echo "spin_*" > /sys/kernel/debug/tracing/set_ftrace_filter
Trace all functions with
cpuin the name:
# echo "cpu" > /sys/kernel/debug/tracing/set_ftrace_filter