5.5. ftrace

The ftrace framework provides users with several tracing capabilities, accessible through an interface much simpler than SystemTap's. This framework uses a set of virtual files in the debugfs file system; these files enable specific tracers. The ftrace function tracer outputs each function called in the kernel in real time; other tracers within the ftrace framework can also be used to analyze wakeup latency, task switches, kernel events, and the like.
You can also add new tracers for ftrace, making it a flexible solution for analyzing kernel events. The ftrace framework is useful for debugging or analyzing latencies and performance issues that take place outside of user-space. Unlike other profilers documented in this guide, ftrace is a built-in feature of the kernel.

5.5.1. Using ftrace

The Red Hat Enterprise Linux 6 kernels have been configured with the CONFIG_FTRACE=y option. This option provides the interfaces required by ftrace. To use ftrace, mount the debugfs file system as follows:
mount -t debugfs nodev /sys/kernel/debug
All the ftrace utilities are located in /sys/kernel/debug/tracing/. View the /sys/kernel/debug/tracing/available_tracers file to find out what tracers are available for your kernel:
cat /sys/kernel/debug/tracing/available_tracers 
power wakeup irqsoff function sysprof sched_switch initcall nop
To use a specific tracer, write it to /sys/kernel/debug/tracing/current_tracer. For example, wakeup traces and records the maximum time it takes for the highest-priority task to be scheduled after the task wakes up. To use it:
echo wakeup > /sys/kernel/debug/tracing/current_tracer
To start or stop tracing, write to /sys/kernel/debug/tracing/tracing_on, as in:
echo 1 > /sys/kernel/debug/tracing/tracing_on (enables tracing)
echo 0 > /sys/kernel/debug/tracing/tracing_on (disables tracing)
The results of the trace can be viewed from the following files:
/sys/kernel/debug/tracing/trace
This file contains human-readable trace output.
/sys/kernel/debug/tracing/trace_pipe
This file contains the same output as /sys/kernel/debug/tracing/trace, but is meant to be piped into a command. Unlike /sys/kernel/debug/tracing/trace, reading from this file consumes its output.