3.9. Latency tracing using trace-cmd

trace-cmd is a MRG Realtime function that traces all kernel function calls, and some special events. It records what is happening in the system during a short period of time, providing information that can be used to analyze system behavior.
The trace-cmd tool is not enabled in the production version of the MRG Realtime kernel as it creates additional overhead. If you wish to use the trace-cmd tool you will need to download and install either the trace or debug variants of the MRG Realtime kernel.

Note

For instructions on how to install kernel variants, see the MRG Realtime Installation Guide.
  1. Once you are using either the trace or debug variants of the MRG Realtime kernel, you can install the trace-cmd tool using yum.
    # yum install trace-cmd
    
  2. To start the utility, type trace-cmd at the shell prompt, along with the options you require, using the following syntax:
    # trace-cmd [command]
    
    The use of the -f option sets Function Tracing and can be used with any other trace command.
    The commands instruct trace-cmd to trace in specific ways.
    Command Description
    record Record a trace into a trace.dat file.
    start Start tracing without recording into a file.
    extract Extract a trace from the kernel.
    stop Stops the kernel from recording trace data.
    reset Disable all kernel tracing and clear the trace buffers.
    report Read out the trace stored in a trace.dat file.
    split Parse a trace.dat file into smaller file(s).
    listen Listen on a network socket for trace clients.
    list List the available events, plugins or options.
    Command Trace Type Description
    -s Context switch Traces the context switches between tasks.
    -i Interrupts off Records the maximum time that an interrupt is disabled. When a new maximum is recorded, it replaces the previous maximum.
    -p Pre-emption off Records the maximum time that pre-emption is disabled. When a new maximum is recorded, it replaces the previous maximum.
    -b Pre-emption and interrupts off Records the maximum time that pre-emption or interrupts are disabled. When a new maximum is recorded, it replaces the previous maximum.
    -w Wakeup Traces and records the maximum time for the highest priority task to get scheduled after it has been woken up.
    -e Event tracing
    -f Function tracing Can be used with any other trace
    -l Prints log in the latency_trace format Can be used with any other trace

    Note

    For further information about event tracing and function tracer refer to Appendix A, Event Tracing and Appendix B, Function Tracer.
  3. In this example, the trace-cmd utility will trace a single trace point:
    # ./trace-cmd record -e sched_wakeup ls /bin