3.10. Latency Tracing Using trace-cmd

trace-cmd is a front-end tool to ftrace. It can enable the ftrace interactions described earlier without needing to write into the /sys/kernel/debug/tracing/ directory. It can be installed without the special tracing kernel variants, and it does not add any overhead when it is installed.
  1. To install the trace-cmd tool, enter the following command as root:
    ~]# yum install trace-cmd
  2. To start the utility, type trace-cmd at the shell prompt, along with the options you require, using the following syntax:
    ~]# trace-cmd command
    Some examples of commands are:
    • ~]# trace-cmd record -p function myapp
      Enable and start recording functions executing within the kernel while myapp runs. It records functions from all CPUS and all tasks, even those not related to myapp.
    • ~]# trace-cmd report
      Display the result.
    • ~]# trace-cmd record -p function -l 'sched*' myapp
      Record only functions that start with sched while myapp runs.
    • ~]# trace-cmd start -e irq
      Enable all the IRQ events.
    • ~]# trace-cmd start -p wakeup_rt
      Start the wakeup_rt tracer.
    • ~]# trace-cmd start -p preemptirqsoff -d
      Start the preemptirqsoff tracer but disable function tracing in doing so. Note: the version of trace-cmd in Red Hat Enterprise Linux 7 turns off ftrace_enabled instead of using the function-trace option. You can enable it again with trace-cmd start -p function.
    • ~]# trace-cmd start -p nop
      Restore the state in which the system was before trace-cmd started modifying it. This is important if you want to use the debugfs file system after using trace-cmd, whether or not the system was restarted in the meantime.


    See the trace-cmd(1) man page for a complete list of commands and options. All the individual commands also have their own man pages, trace-cmd-command. For further information about event tracing and function tracer, see Appendix A, Event Tracing and Appendix B, Detailed Description of Ftrace.
  3. In this example, the trace-cmd utility will trace a single trace point:
    ~]# trace-cmd record -e sched_wakeup ls /bin