3.8. Latency Tracing Using trace-cmd

trace-cmd is a MRG Realtime function that traces all kernel function calls, and some special events. It records what is happening in the system during a short period of time, providing information that can be used to analyze system behavior.
The trace-cmd tool is not enabled in the production version of the MRG Realtime kernel as it creates additional overhead. If you wish to use the trace-cmd tool you will need to download and install either the trace or debug variants of the MRG Realtime kernel.

Note

For instructions on how to install kernel variants, see the MRG Realtime Installation Guide.
  1. Once you are using either the trace or debug variants of the MRG Realtime kernel, you can install the trace-cmd tool using yum.
    # yum install trace-cmd
    
  2. Mount the kernel to /sys/kernel/debug and instruct it to use the debugfs file system.
    # mount -t debugfs nodev /sys/kernel/debug
    
  3. You can choose to make the debugfs directory mount automatically on boot. You can do this by opening the /etc/fstab file in your preferred text editor, and adding the following line:
    nodev /sys/kernel/debug	   debugfs   defaults   0  0
    
  4. To start the utility, type trace-cmd at the shell prompt, along with the options you require, using the following syntax:
    # trace-cmd [-f] [command]
    
    The use of the -f option sets Function Tracing and can be used with any other trace command.
    The commands instruct trace-cmd to trace in specific ways.
    Command Trace Type Description
    -s Context switch Traces the context switches between tasks.
    -i Interrupts off Records the maximum time that an interrupt is disabled. When a new maximum is recorded, it replaces the previous maximum.
    -p Pre-emption off Records the maximum time that pre-emption is disabled. When a new maximum is recorded, it replaces the previous maximum.
    -b Pre-emption and interrupts off Records the maximum time that pre-emption or interrupts are disabled. When a new maximum is recorded, it replaces the previous maximum.
    -w Wakeup Traces and records the maximum time for the highest priority task to get scheduled after it has been woken up.
    -e Event tracing
    -f Function tracing Can be used with any other trace
    -l Prints log in the latency_trace format Can be used with any other trace
  5. In this example, the trace-cmd utility will:
    • Select the context switch tracing method
    • Enable the latency tracer
    • Run the ls -la command
    • Turn the latency tracer off again
    # trace-cmd -s /bin/ls -la > /tmp/latency_log.txt